Raspberry π

Posted on
Page
of 53
  • Ah right, t'other page said json for sure, and looking again gives example.

    Reading your link "gtfs" might be better to use? Fuck knows now, feels like I'm starting all over again.

  • It might need you to specify the Accept: application/json header to tell the server you definitely want the response in JSON.

    Vaguely related: https://techforum.tfl.gov.uk/t/json-vs-xml-random-content-type-in-response/2238/13

  • Can you try (based on https://stackoverflow.com/questions/8685790/adding-headers-to-requests-module):-

    import requests
    headers = {'Accept': 'application/json'}
    url = 'https://data.bus-data.dft.gov.uk/api/v1­/datafeed/1695/?api_key=***'
    response = requests.get(url, headers=headers)
    ...
    
  • Reading your link "gtfs" might be better to use?

    I think GTFS is a specification the transport companies should stick to when publishing this data, not a different way of accessing the data, if that makes sense?
    So when the initial doc page said "Results are returned as JSON.", it was wrong. Really, the data adheres to GTFS and is therefore going to be .xml. Unless maybe you force it to be json like greenbank is saying and I was garbling through on the last page

  • Not sure where I'd stick that line into the code? It sounds like it isn't alternating though, is just always xml.

    Thanks though I'm learning some things this evening just not along the path I'd hoped to progress on.

  • Ah, I'll try this ta.

  • The two lines I've given just replace the existing requests.get line. [EDIT] Copied the code from the wrong example so I've updated it now.

    The link to the alternating problem was to highlight that it "should" change its behaviour based on the value of the Accept header, albeit with some bug in the server's implementation.

    With no header provided (as your existing code does) it may default to XML.

    It may appear as JSON when viewed in the browser because the browser is providing the accept header set with it set to (amongst other things) application/json.

  • I think that makes sense,although a quick m Google gives info on gtfs -> json conversion?

    It's a whole new day of learning I think and I can usually focus on this only once every few months at best 🤔

  • Ok the type is now

    text/xml; charset=utf-8
    

    Which is different

    Status code was 406 which is a fail. Progress though maybe?

  • 406 is "not available in preferred format"

  • I think xmltodict is the solution, again my thought process & application hampered by examples being local file based with maybe just 1 example data row and only a few headings which my pea-brain can't quickly adapt to what I'd like to do. Running before I can walk, I know.

  • Thanks both @yoshy & @Greenbank, it has been useful / humbling / exasperating / humiliating in equal measure for me, you've both been very helpful on the subject since I posted.

    Hope you got your budget done too @yoshy

  • I found more info, xmltodict does a conversion but I need to get my ahead around keys now it seems? I can get it to show the top level but no further, looks like 7 or 8 levels via the url.

    import requests 
    
    import xmltodict
    
    url = 'https://data.bus-data.dft.gov.uk/api/v1/datafeed/1695/?api_key=***'
    
    response = requests.get(url)
    
    r = xmltodict.parse(response.content)
    
    [#print](https://www.lfgss.com/search/?q=%23print)(r) [#useful](https://www.lfgss.com/search/?q=%23useful) check line, showed a more expected json like output
    
    print("Status code:", response.status_code)
    
    print(response.headers['Content-Type'])
    
    result = list(r.keys())
    
    print(result)
    
    

    Gives no error:

    Status code: 200
    
    text/xml
    
    ['Siri']
    
    
    

    I've no clue how to get it so I can filter by bus service number & direction from here, but that's the plan. Step 1 will be for those fields to show on or after line 3 of the output I guess.

  • Do a bit of reading about dictionaries, the general idea being that you have key+value pairs.
    If you tell the program:
    print(r['Siri'])
    You're saying "get me the value from the datastructure called 'r' which is associated with the key 'Siri'"
    Similar to if you have an array called arr and want the first value you'll say arr[0].

    To get further into the file, try adding another set of square brackets:
    print(r['Siri'] ['ServiceDelivery'])

    This might work or I could be making an arse of myself, let's see!

  • Thanks, I will read up. C&P of

    print(r['Siri'])

    Gives the full output (all live busses) so it's picking something up.

    print(r['Siri'] ['ServiceDelivery'])

    Gives the same, basically every live bus.

    If I can get a filter in place I can work with a single LineRef (bus number) which means much less info coming back, that's where reading about dictionaries could/should help.

  • So I was just using the screenshot you posted last page to start indexing into the data, using the indents in the file to determine the 'level' of indexing needed. The next level looks like it'll be 'VehicleMonitoringDelivery', within which there are at least 2 'VehicleActivity'.
    This is where things get a bit more involved... a key in a dictionary has to be unique, so the xmltodict library is going to have to do some jiggery pokery to make it work (put the VehicleActivity elements in an array: https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html (I think!))

    Thanks for posting about this btw, it's inspired me to get my dev environment set up on my pc and get noodling again.

  • Also please do tell me to shut up if you just want to work away and suss things out on your own

  • the xmltodict library is going to have to do some jiggery pokery to make it work

    I bet £5 there will be a package to do this for you...

  • The very first example in the xmltodict docs page has two <many> keys and they're concatenated into a list. So hopefully those VehicleActivity items will be fine.

  • Yeah sorry that's what I meant about the array in my post, my bad

  • I'm well out of my depth already, so guidance that helps me understand more and stay afloat or get closer to be able to figuring things out in future is useful thanks.

  • Anyone got any experience with NAS cases for a 3b+ beyond the Argon EON?

    Is it even worth it without USB 3.0 bus? Which it seems like all Pi based NASs use.

  • I put circuit python on a RPi Pico, and then using the HID library I got it to pretend to be a mouse when plugged into a computer.

    But when it's plugged into a computer, it also appears as a mass storage device (that's how you update the python code).

    So how is it appearing at two devices? I assumed it was setting up a virtual USB hub and "attaching" two devices to it, but that doesn't seem the be the case. When I run lsusb I only see one device. I'm so confused

  • Did a little bit of googling and I'm not claiming this to be definite but I think the one device is opening up two communication channels with the PC over the 1 USB connection. 1 of those comms channels is just having fairly simple generic USB HID messages sent over it and the pc is accepting these and saying "okay there's a mouse there" and on the other comms channels it's sent whatever it needs to to say "this is a generic storage device, you can mount this and use it".
    If you run "ls /dev" without it plugged in, then again with it plugged in, how many more devices does it show? Out of interest

  • Oh well, my twitter timelapse project dies on 9th Feb, along with many other cooler & more useful accounts no doubt

    https://twitter.com/TwitterDev/status/1621026986784337922

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Raspberry π

Posted by Avatar for photoben @photoben

Actions