Guide to DevSecOps

Appearance

Comprehensive Guide to Python Project Development with Poetry ​

This guide walks you through creating, developing, and publishing a Python project using Poetry—a modern dependency management and packaging tool.

1. Introduction to Poetry ​

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and manages them for you.

Why Choose Poetry? ​

  • Simplified dependency management: Handles dependencies and subdependencies
  • Isolated environments: Automatically creates and manages virtual environments
  • Consistent builds: Lock files ensure reproducible installations
  • Modern packaging: Simplifies building and publishing packages

2. Installation ​

Installing Poetry ​

bash
# Install Poetry using the official installer
curl -sSL https://install.python-poetry.org | python3 -

# Verify installation
poetry --version

Configure Poetry ​

bash
# Configure Poetry to create virtual environments in the project directory
poetry config virtualenvs.in-project true

# Configure other settings as needed
poetry config --list

3. Creating a New Project ​

Start a new project ​

bash
# Create a new project
poetry new my-awesome-project
cd my-awesome-project

This creates a standard project structure:

my-awesome-project/
├── pyproject.toml       # Project configuration
├── README.md            # Project documentation
├── my_awesome_project/  # Source code
│   └── __init__.py      # Package initialization
└── tests/               # Test directory
    └── __init__.py      # Test initialization

Understanding pyproject.toml ​

The pyproject.toml file is the core of your project configuration:

toml
[tool.poetry]
name = "my-awesome-project"
version = "0.1.0"
description = "An amazing Python project"
authors = ["Your Name <your.email@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]
pytest = "^7.0.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

4. Managing Dependencies ​

Adding Production Dependencies ​

bash
# Add required packages
poetry add requests
poetry add pandas numpy matplotlib

# Add package with version constraint
poetry add flask@^2.0.0

Adding Development Dependencies ​

bash
# Add development dependencies
poetry add --group dev pytest pytest-cov black flake8
poetry add --group dev mypy

Removing Dependencies ​

bash
# Remove a package
poetry remove requests

Viewing Dependencies ​

bash
# List all dependencies
poetry show

# View dependency tree
poetry show --tree

5. Managing Virtual Environments ​

Using Poetry's Environment ​

bash
# Activate the virtual environment
poetry shell

# Run commands in the virtual environment without activation
poetry run python -V

Installing Dependencies ​

bash
# Install dependencies from pyproject.toml
poetry install

# Install without development dependencies
poetry install --no-dev

6. Project Development ​

Structuring Your Project ​

Expand your package structure:

my-awesome-project/
├── pyproject.toml
├── README.md
├── my_awesome_project/
│   ├── __init__.py
│   ├── core.py         # Core functionality
│   ├── utils.py        # Utility functions
│   └── cli.py          # Command-line interface
├── tests/
│   ├── __init__.py
│   ├── test_core.py    # Tests for core.py
│   └── test_utils.py   # Tests for utils.py
└── docs/               # Documentation
    └── index.md        # Main documentation file

Creating Entry Points ​

Add command-line scripts to pyproject.toml:

toml
[tool.poetry.scripts]
my-command = "my_awesome_project.cli:main"

Writing Tests ​

Create test files in the tests directory:

python
# tests/test_core.py
import pytest
from my_awesome_project import core

def test_sample_function():
    result = core.sample_function()
    assert result == "expected output"

Running Tests ​

bash
# Run tests with Poetry
poetry run pytest

# With coverage
poetry run pytest --cov=my_awesome_project

7. Code Quality Tools ​

Configure Black and Flake8 ​

Create configuration files:

toml
# pyproject.toml (for Black)
[tool.black]
line-length = 88
target-version = ["py38"]
include = '\.pyi?$'

[tool.isort]
profile = "black"
ini
# .flake8
[flake8]
max-line-length = 88
extend-ignore = E203
exclude = .git,__pycache__,build,dist

Running Code Quality Tools ​

bash
# Format code with Black
poetry run black my_awesome_project tests

# Check imports with isort
poetry run isort my_awesome_project tests

# Lint with flake8
poetry run flake8 my_awesome_project tests

# Type check with mypy
poetry run mypy my_awesome_project

8. Documentation ​

Setting Up Documentation ​

bash
# Add mkdocs for documentation
poetry add --group dev mkdocs mkdocs-material

# Initialize documentation
poetry run mkdocs new .

Building Documentation ​

bash
# Build documentation
poetry run mkdocs build

# Serve documentation locally
poetry run mkdocs serve

9. Building Your Package ​

bash
# Build source and wheel distributions
poetry build

# This creates distribution files in dist/
# - my_awesome_project-0.1.0.tar.gz (source)
# - my_awesome_project-0.1.0-py3-none-any.whl (wheel)

10. Publishing Your Package ​

Configure PyPI Credentials ​

bash
# Add PyPI token (recommended)
poetry config pypi-token.pypi your-pypi-token

# Or configure username/password
poetry config http-basic.pypi username password

Publish to PyPI ​

bash
# Publish to PyPI
poetry publish

# Build and publish in one command
poetry publish --build

# Publish to TestPyPI first (good practice)
poetry publish -r test-pypi

Publishing to Private Repositories ​

Configure a custom repository in pyproject.toml:

toml
[[tool.poetry.source]]
name = "private"
url = "https://private-repo.example.com/simple/"
default = false
secondary = false

Then publish to it:

bash
poetry publish -r private

11. Version Management ​

bash
# Bump version (patch, minor, major, etc.)
poetry version patch  # 0.1.0 -> 0.1.1
poetry version minor  # 0.1.1 -> 0.2.0
poetry version major  # 0.2.0 -> 1.0.0

# Custom version
poetry version 2.3.4

12. Continuous Integration ​

Example GitHub Actions Workflow ​

Create .github/workflows/python-package.yml:

yaml
name: Python Package

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.8, 3.9, 3.10]

    steps:
      - uses: actions/checkout@v2

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install Poetry
        uses: snok/install-poetry@v1
        with:
          virtualenvs-create: true
          virtualenvs-in-project: true

      - name: Install dependencies
        run: poetry install

      - name: Lint with flake8
        run: poetry run flake8

      - name: Format check with black
        run: poetry run black --check .

      - name: Test with pytest
        run: poetry run pytest --cov

13. Advanced Poetry Usage ​

Managing Multiple Python Versions ​

Use pyenv alongside Poetry:

bash
# Install Python versions with pyenv
pyenv install 3.8.12
pyenv install 3.9.9

# Set local Python versions
pyenv local 3.8.12 3.9.9

# Poetry will use pyenv's Python versions

Poetry Environment Management ​

bash
# List all environments
poetry env list

# Remove an environment
poetry env remove python3.8

# Use specific Python interpreter
poetry env use /path/to/python

Export Requirements ​

bash
# Export to requirements.txt
poetry export -f requirements.txt --output requirements.txt

# Include development dependencies
poetry export -f requirements.txt --with dev --output dev-requirements.txt

14. Best Practices ​

  1. Always use lock files: Commit poetry.lock to ensure reproducible builds
  2. Group dependencies logically: Use dependency groups for dev, test, docs
  3. Keep dependencies minimal: Only add what you need
  4. Pin versions appropriately: Use ^ for flexible but safe versions
  5. Document your package: Include usage examples in README
  6. Use semantic versioning: Follow the major.minor.patch convention
  7. Add type hints: Improve code quality with type annotations
  8. Automate with CI/CD: Set up automated testing and publishing

Conclusion ​

Poetry streamlines Python project development from initialization to publication. By following this guide, you've learned how to create a well-structured project, manage dependencies, ensure code quality, create documentation, and publish your package to PyPI.

The modern workflow with Poetry enables you to focus more on coding and less on package management complexity.