2 Replies Latest reply on Nov 8, 2016 4:35 AM by parasg

    Python script against the API to pull user data

    mpetrosky

      Hi, I'm a novice python programmer and I wrote this to extract all of our users and their point balance.  The problem I'm running into is that this winds up pulling duplicate records randomly throughout the results.  When I look at the data, it is almost like the API returns a duplicate page of results a couple times during the extract.

       

      I wrote the pagination loop myself, so I'm curious if any other python gurus out there have cracked this a different way?

       

      import requests, base64, csv
      
      # jive api pagination variables
      startIndex = 0
      itemsPerPage = 25
      # jive api username
      username = 'blablabla'
      password = 'blablabla'
      
      headers = {'Authorization': 'basicAuthKey'}
      
      with open('c:\users\xxx\documents\y_points.csv', 'wb') as csvfile:
        writer = csv.writer(csvfile, delimiter=',')
         while True:
         # reset the pagination counter, if we get a page with fewer than 25 items, we've reached the end
         counter = 0
         # variables to build the url
         url = 'https://community.community.com/api/core/v3/people'
         filters = '?filter=include-disabled'
         count = '&count=' + str(itemsPerPage)
        index = '&startIndex=' + str(startIndex)
         # api root access point using url variables
         rest_data = requests.get(url + filters + count + index, auth=(username, password), headers=headers)
         # convert the return into a json blob
         blob = rest_data.json()
      
         # parse the blob to pull out users and points
         for x in blob['list']:
        points = x['jive']['level']['points']
        name = x['displayName']
        user_id = x['jive']['username']
        counter += 1
         #write the values to the csv file
         writer.writerow([name.encode('utf-8'), str(points), user_id.encode('utf-8')])
      
         # increment the starting point for the next page
         startIndex += blob['itemsPerPage']
      
         # test to see if we've reached the end of the query
         if counter < 25:
      break
      

       

        • Re: Python script against the API to pull user data
          pawans

          I had writen few python 2.7 scripts for jive .

          Here's my github repo --> GitHub - shahpawan/jive-python-scripts: Python scripts to perform several admin tasks on jive via jive rest apis

           

          I have not updated this repo since several months. But most of it should still work.

          • Re: Python script against the API to pull user data
            parasg

            Your code was working fine.

             

            When we were using the same API to pull user data, there were some missing records instead of duplicate ones. Sometimes API returns 24 records instead of 25.

            I think that might be issue with Jive API.

             

            There are few alterations done in the code below.

            - Instead of incrementing the "startIndex" value, Jive JSON returns "next" fields in "link" which contains the next API url which can be used to pull next page data.

             

             

            import requests, base64, csv, json  
            
            
            # jive api pagination variables  
            startIndex = 0  
            itemsPerPage = 25  
            # jive api username  
            username = 'blablabla'  
            password = 'blablabla'  
            
            
            headers = {'Authorization': 'basicAuthKey'}  
            url = 'https://community.community.com/api/core/v3/people?filter=include-disabled'
            with open('c:\users\xxx\documents\y_points.csv', 'wb') as csvfile:  
                writer = csv.writer(csvfile, delimiter=',')  
                while True:  
                    rest_data = requests.get(url, auth=(username, password), headers=headers)  
                    # convert the return into a json blob  
                    blob = rest_data.json()
                    # parse the blob to pull out users and points  
                    for x in blob['list']:  
                        # points = (('level' in x['jive']) and ('points' in x['jive']['level']) and x['jive']['level']['points']) or '--'  
                        points = x['jive']['level']['points']  
                        name = x['displayName']  
                        user_id = x['jive']['username']  
                        #write the values to the csv file  
                        writer.writerow([name.encode('utf-8'), str(points), user_id.encode("utf-8")])  
            
            
                    # test to see if we've reached the end of the query  
                    if ('links' in blob) and ('next' in blob['links']): 
                        url = blob['links']['next'];
                    else:
                        break;
            
            2 people found this helpful