2 Replies Latest reply on Nov 8, 2016 4:35 AM by parasg

    Python script against the API to pull user data

    mpetrosky
    mpetrosky Nov 7, 2016 12:41 PM

      Hi, I'm a novice python programmer and I wrote this to extract all of our users and their point balance.  The problem I'm running into is that this winds up pulling duplicate records randomly throughout the results.  When I look at the data, it is almost like the API returns a duplicate page of results a couple times during the extract.

       

      I wrote the pagination loop myself, so I'm curious if any other python gurus out there have cracked this a different way?

       

      import requests, base64, csv

# jive api pagination variables
startIndex = 0
itemsPerPage = 25
# jive api username
username = 'blablabla'
password = 'blablabla'

headers = {'Authorization': 'basicAuthKey'}

with open('c:\users\xxx\documents\y_points.csv', 'wb') as csvfile:
  writer = csv.writer(csvfile, delimiter=',')
   while True:
   # reset the pagination counter, if we get a page with fewer than 25 items, we've reached the end
   counter = 0
   # variables to build the url
   url = 'https://community.community.com/api/core/v3/people'
   filters = '?filter=include-disabled'
   count = '&count=' + str(itemsPerPage)
  index = '&startIndex=' + str(startIndex)
   # api root access point using url variables
   rest_data = requests.get(url + filters + count + index, auth=(username, password), headers=headers)
   # convert the return into a json blob
   blob = rest_data.json()

   # parse the blob to pull out users and points
   for x in blob['list']:
  points = x['jive']['level']['points']
  name = x['displayName']
  user_id = x['jive']['username']
  counter += 1
   #write the values to the csv file
   writer.writerow([name.encode('utf-8'), str(points), user_id.encode('utf-8')])

   # increment the starting point for the next page
   startIndex += blob['itemsPerPage']

   # test to see if we've reached the end of the query
   if counter < 25:
break

       

      • 1319 Views
      • Tags:
        • Re: Python script against the API to pull user data
          pawans
          pawans Nov 7, 2016 9:44 PM (in response to mpetrosky)

          I had writen few python 2.7 scripts for jive .

          Here's my github repo --> GitHub - shahpawan/jive-python-scripts: Python scripts to perform several admin tasks on jive via jive rest apis

           

          I have not updated this repo since several months. But most of it should still work.

          • Re: Python script against the API to pull user data
            parasg
            parasg Nov 8, 2016 4:35 AM (in response to mpetrosky)

            Your code was working fine.

             

            When we were using the same API to pull user data, there were some missing records instead of duplicate ones. Sometimes API returns 24 records instead of 25.

            I think that might be issue with Jive API.

             

            There are few alterations done in the code below.

            - Instead of incrementing the "startIndex" value, Jive JSON returns "next" fields in "link" which contains the next API url which can be used to pull next page data.

             

             

            import requests, base64, csv, json  


# jive api pagination variables  
startIndex = 0  
itemsPerPage = 25  
# jive api username  
username = 'blablabla'  
password = 'blablabla'  


headers = {'Authorization': 'basicAuthKey'}  
url = 'https://community.community.com/api/core/v3/people?filter=include-disabled'
with open('c:\users\xxx\documents\y_points.csv', 'wb') as csvfile:  
    writer = csv.writer(csvfile, delimiter=',')  
    while True:  
        rest_data = requests.get(url, auth=(username, password), headers=headers)  
        # convert the return into a json blob  
        blob = rest_data.json()
        # parse the blob to pull out users and points  
        for x in blob['list']:  
            # points = (('level' in x['jive']) and ('points' in x['jive']['level']) and x['jive']['level']['points']) or '--'  
            points = x['jive']['level']['points']  
            name = x['displayName']  
            user_id = x['jive']['username']  
            #write the values to the csv file  
            writer.writerow([name.encode('utf-8'), str(points), user_id.encode("utf-8")])  


        # test to see if we've reached the end of the query  
        if ('links' in blob) and ('next' in blob['links']): 
            url = blob['links']['next'];
        else:
            break;
            2 people found this helpful