How to get list of jobs, runs, and then artifacts?

[wassimj]

Hi,
With Speckle, I can authenticate a user, then get their streams, they can choose a stream from the list, the app then lists their branches, the user chooses a branch, the app lists their commits, and finally they can choose a commit. I would like to do the same with pollination from streamlit. Mainly:

1. The user is authenticated with an api token
2. The app displays a list of their projects
3. The user chooses a project from the list of projects
4. The app displays a list of jobs
5. The user chooses a job from the above list
6. The app displays a list of runs
7. The user chooses a run from the above list
8. The app displays a list of downloadable artifacts
9. The user chooses to download a specific artifact from the above list

What is the skeleton python code to make this happen? Thanks!

[mostapha]

Hi, @wassimj! Have you seen the tutorials here?

Pollination Cloud App

It walks you through the process that you’re trying to create.

[wassimj]

Don’t know how I missed that I had tried those before! Thanks. Exactly what I needed

[wassimj]

The tutorial is missing the very last step. How to download artifacts. Other tutorials include code for that, but are very confusing compared to this. There also seems to be confusion in terminology between owner, account, user?
It would be great if the above tutorials included how to download a particular artifact from a run. Thank you

[wassimj]

I got to the point of downloading an artifact from a run, but I get this error

AttributeError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you’re on Streamlit Cloud, click on ‘Manage app’ in the lower right of your app).

Traceback:

File "/home/appuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 556, in _run_script
    exec(code, module.__dict__)File "/app/topologichb/pages/04_download.py", line 59, in <module>
    api_response = api_instance.download_run_artifact(account['username'], project['name'], run['id'], path=path_to_file)File "/home/appuser/venv/lib/python3.9/site-packages/pollination_sdk/api/runs_api.py", line 245, in download_run_artifact
    return self.download_run_artifact_with_http_info(owner, name, run_id, **kwargs)  # noqa: E501File "/home/appuser/venv/lib/python3.9/site-packages/pollination_sdk/api/runs_api.py", line 315, in download_run_artifact_with_http_info
    if self.api_client.client_side_validation and ('owner' not in local_var_params or  # noqa: E501

[nicolas]

@wassimj - you’re a little ahead of us! As is appropriate

We’re still working on “closing the loop” with job status / job artifacts components.

Here’s a python implementation. It’s a little streamlit app, so you might have to pick out the parts that are relevant to your use case.

pollination-apps/artifacts-example/blob/main/app/app.py

[Edited, removed original example in favor of new reference example]

[nicolas]

I deployed this little app here:

Pollination Cloud App

One thing I realized is that the app throws an error if it tries to decode a .zip file (I believe). That’s more with the decoding logic than the downloading an artifact logic though.

[wassimj]

Thank you. Will try this code. It would be very helpful if all these libraries are explained somewhere. I am getting confused by interactors and selectors job vs study etc. I think these are probably part of growing pains.

[wassimj]

@nicolas

api_client = get_api_client()

job = job_selector(api_client)

def handleSelectArtifact():
    bytes = st.session_state['sel_artifact'].download().read()
    st.session_state['text'] = bytes.decode('utf8')

def formatArtifact(artifact):
    if 'key' in artifact.__dict__:
        return artifact.key
    else:
        return ''

if 'text' not in st.session_state:
    st.session_state['text'] = ''

if 'artifacts' not in st.session_state:
    st.session_state['artifacts'] = []

def followArtifactTree(artifact_array):
    for artifact in artifact_array:
        if artifact.is_folder:
            followArtifactTree(artifact.list_children())
        else :
              st.session_state['artifacts'].append(artifact)
    
if job is not None:
    artifacts = job.list_artifacts()

    if artifacts is not None:
        followArtifactTree(artifacts)

st.selectbox(
  'Select an artifact', 
  options= st.session_state['artifacts'], 
  key='sel_artifact', 
  on_change=handleSelectArtifact, 
  format_func=formatArtifact
)

st.download_button(
    label='Download Text File'

Thanks. Yes it gives me an error immediately: Error: Invalid base64-encoded string: number of data characters (2245) cannot be 1 more than a multiple of 4

[nicolas]

Hey @wassimj - apologies. That is a really weird error because we thought we had resolved it a while ago.

I am working on a better solution for you and will have it up later today.

Yes, the lack of documentation is due to the fact that these libraries are still under development.

The solution I’m working on will rely on pollination-streamlit-io which should be more straightforward. The python library is more of throwback to when we were first experimenting with Streamlit.

Thanks for your patience!

[mostapha]

@wassimj

I am getting confused by interactors and selectors job vs study etc. I think these are probably part of growing pains.

This is a fair comment, and I think you will like what @nicolas is putting together with pollination-streamlit-io. That will abstract the complexities of the API for you.

That said, I agree that we need better documentation on the core libraries themselves.

[nicolas]

@wassimj -

Here’s a better implementation using pollination-streamlit-io:

pollination-apps/artifacts-example/blob/main/app/app.py

And I’ve redeployed it to the same url:

Pollination Cloud App

(It should finish deploying shortly)

Please let me know if this matches your use case or if you need further assistance.