PyPI and Anaconda maintainer workflow using pipreqs, flit, grayskull, and conda-souschef

[sgbaird]

Figured it might be better to have its own issue, even though it's not really an "issue" perse (#29).

It's entirely possible that this workflow is overkill or misguided, esp. since I just started learning PyPI and Anaconda building and maintaining within the last few weeks or so. There are some rough GitHub actions workflows at mat_discover/.github/workflows. It would be great to hear your thoughts/suggestions.

dependencies

conda config --add channels conda-forge
conda config --set anaconda_upload yes
conda install pipreqs flit grayskull anaconda-client

setup
(see https://flit.readthedocs.io/en/latest/ for flit instructions, minimally discussed in following comments)

pipreqs . # outputs requirements.txt based on imports
# <touch up requirements.txt>
flit init
# <touch up pyproject.toml>
# <add `__init__.py` under `mat_discover/mat_discover` with docstring and e.g. `__version__ = "1.3.9"`>
# <add `dependencies = ["", "", "", ...]` to pyproject.toml based on requirements.txt>
# <add PyPI credentials (i.e. token) to `~/.pypirc`>
grayskull pypi mat_discover
# <touch up run_grayskull.py based on default output of grayskull>

manually modified run_grayskull.py

"""Touch up the conda recipe from grayskull using conda-souschef."""
from souschef.recipe import Recipe
import os
from os.path import join
from pathlib import Path
import mat_discover

name, version = mat_discover.__name__, mat_discover.__version__
os.system("grayskull pypi {0}=={1}".format(name, version))

Path("scratch").mkdir(exist_ok=True)

fpath = join(name, "meta.yaml")
fpath2 = join("scratch", "meta.yaml")
my_recipe = Recipe(load_file=fpath)
#--touchups---
my_recipe["requirements"]["host"].append("flit")
my_recipe["build"]["noarch"] = "python"
del my_recipe["requirements"]["build"]
my_recipe["requirements"]["run"].replace("kaleido", "python-kaleido")
my_recipe["requirements"]["run"].append("pytorch >=1.9.0")
my_recipe["requirements"]["run"].append("cudatoolkit <11.4")
#^^touchups^^
my_recipe.save(fpath)
my_recipe.save(fpath2)

publish to PyPI and Anaconda

flit publish 
python run_grayskull.py
cd scratch
conda build .

Initial grayskull output:

{% set name = "mat_discover" %}
{% set version = "1.3.9" %}


package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/mat_discover-{{ version }}.tar.gz
  sha256: 082b815cb74caeb1d2475be206f2e6779981b05ca85b81b6fbe4dcee3e616211

build:
  number: 0
  script: {{ PYTHON }} -m pip install . -vv

requirements:
  build:
    - {{ compiler('c') }}
  host:
    - pip
    - python
  run:
    - chem_wasserstein >=1.0.6
    - crabnet >=1.0.6
    - cython
    - dill
    - hdbscan
    - kaleido
    - matplotlib-base >=3.4.3
    - numba >=0.53.1
    - pandas
    - plotly
    - pqdm
    - python
    - scikit-learn
    - scipy
    - seaborn
    - tqdm
    - umap-learn

test:
  imports:
    - mat_discover
  commands:
    - pip check
  requires:
    - pip

about:
  home: https://pypi.org/project/mat_discover/
  summary: Data-driven materials discovery based on composition.
  dev_url: https://github.com/sparks-baird/mat_discover
  license: GPL-3.0
  license_file: LICENSE

extra:
  recipe-maintainers:
    - sgbaird

final grayskull output:
Note: several of these packages come from the sgbaird Anaconda channel

{% set name = "mat_discover" %}
{% set version = "1.3.9" %}


package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/mat_discover-{{ version }}.tar.gz
  sha256: 082b815cb74caeb1d2475be206f2e6779981b05ca85b81b6fbe4dcee3e616211

build:
  number: 0
  script: {{ PYTHON }} -m pip install . -vv

  noarch: python
requirements:
  host:
    - pip
    - flit
    - python
  run:
    - chem_wasserstein >=1.0.6
    - cudatoolkit <11.4
    - pytorch >=1.9.0
    - python-kaleido
    - crabnet >=1.0.6
    - cython
    - dill
    - hdbscan
    - matplotlib-base >=3.4.3
    - numba >=0.53.1
    - pandas
    - plotly
    - pqdm
    - python
    - scikit-learn
    - scipy
    - seaborn
    - tqdm
    - umap-learn

test:
  imports:
    - mat_discover
  commands:
    - pip check
  requires:
    - pip

about:
  home: https://pypi.org/project/mat_discover/
  summary: Data-driven materials discovery based on composition.
  dev_url: https://github.com/sparks-baird/mat_discover
  license: GPL-3.0
  license_file: LICENSE

extra:
  recipe-maintainers:
    - sgbaird

To just install instead of upload, you can use:

flit install --pth-file # (similar to `pip install -e .`)
python run_grayskull.py # assuming already on PyPI, or modify within: `grayskull pypi <github-url>`
conda config --set anaconda_upload no
cd scratch
conda build .
conda install --use-local

[sgbaird]

conda/grayskull#254
conda/grayskull#243
pypa/flit#461