<!-- Generated mechanically from Architecture Patterns with Python AsciiDoc source. -->
<!-- Source file: appendix_project_structure.asciidoc. Do not edit book text here; regenerate from upstream if needed. -->

## A Template Project Structure

Around chapter_04_service_layer, we moved from just having
everything in one folder to a more structured tree, and we thought it might
be of interest to outline the moving parts.

> [!TIP]
The code for this appendix is in the
appendix_project_structure branch https://oreil.ly/1rDRC[on GitHub]:

```
git clone https://github.com/cosmicpython/code.git
cd code
git checkout appendix_project_structure
```


The basic folder structure looks like this:

.Project tree
```text
.
├── Dockerfile  <1>
├── Makefile  <2>
├── README.md
├── docker-compose.yml  <1>
├── license.txt
├── mypy.ini
├── requirements.txt
├── src  <3>
│   ├── allocation
│   │   ├── __init__.py
│   │   ├── adapters
│   │   │   ├── __init__.py
│   │   │   ├── orm.py
│   │   │   └── repository.py
│   │   ├── config.py
│   │   ├── domain
│   │   │   ├── __init__.py
│   │   │   └── model.py
│   │   ├── entrypoints
│   │   │   ├── __init__.py
│   │   │   └── flask_app.py
│   │   └── service_layer
│   │       ├── __init__.py
│   │       └── services.py
│   └── setup.py  <3>
└── tests  <4>
    ├── conftest.py  <4>
    ├── e2e
    │   └── test_api.py
    ├── integration
    │   ├── test_orm.py
    │   └── test_repository.py
    ├── pytest.ini  <4>
    └── unit
        ├── test_allocate.py
        ├── test_batches.py
        └── test_services.py
```

<1> Our _docker-compose.yml_ and our _Dockerfile_ are the main bits of configuration
    for the containers that run our app, and they can also run the tests (for CI).  A
    more complex project might have several Dockerfiles, although we've found that
    minimizing the number of images is usually a good idea.footnote:[Splitting
    out images for production and testing is sometimes a good idea, but we've tended
    to find that going further and trying to split out different images for
    different types of application code (e.g., Web API versus pub/sub client) usually
    ends up being more trouble than it's worth; the cost in terms of complexity
    and longer rebuild/CI times is too high. YMMV.]

<2> A __Makefile__ provides the entrypoint for all the typical commands a developer
    (or a CI server) might want to run during their normal workflow: `make
    build`, `make test`, and so on.footnote:[A pure-Python alternative to Makefiles is
    http://www.pyinvoke.org[Invoke], worth checking out if everyone on your
    team knows Python (or at least knows it better than Bash!).] This is optional. You could just use
    `docker-compose` and `pytest` directly, but if nothing else, it's nice to
    have all the "common commands" in a list somewhere, and unlike
    documentation, a Makefile is code so it has less tendency to become out of date.

<3> All the source code for our app, including the domain model, the
    Flask app, and infrastructure code, lives in a Python package inside
    _src_, (https://hynek.me/articles/testing-packaging["Testing and Packaging") by Hynek Schlawack provides more information on _src_ folders.]
    which we install using `pip install -e` and the _setup.py_ file.  This makes
    imports easy. Currently, the structure within this module is totally flat,
    but for a more complex project, you'd expect to grow a folder hierarchy
    that includes _domain_model/_, _infrastructure/_, _services/_, and _api/_.


<4> Tests live in their own folder. Subfolders distinguish different test
    types and allow you to run them separately.  We can keep shared fixtures
    (_conftest.py_) in the main tests folder and nest more specific ones if we
    wish. This is also the place to keep _pytest.ini_.



> [!TIP]
> The https://oreil.ly/QVb9Q[pytest docs] are really good on test layout and importability.


Let's look at a few of these files and concepts in more detail.



### Env Vars, 12-Factor, and Config, Inside and Outside Containers

The basic problem we're trying to solve here is that we need different
config settings for the following:

- Running code or tests directly from your own dev machine, perhaps
  talking to mapped ports from Docker containers

- Running on the containers themselves, with "real" ports and hostnames

- Different container environments (dev, staging, prod, and so on)

Configuration through environment variables as suggested by the
https://12factor.net/config[12-factor manifesto] will solve this problem,
but concretely, how do we implement it in our code and our containers?


### Config.py

Whenever our application code needs access to some config, it's going to
get it from a file called __config.py__. Here are a couple of examples from our
app:

.Sample config functions (src/allocation/config.py)
```python
import os


def get_postgres_uri():  #<1>
    host = os.environ.get("DB_HOST", "localhost")  #<2>
    port = 54321 if host == "localhost" else 5432
    password = os.environ.get("DB_PASSWORD", "abc123")
    user, db_name = "allocation", "allocation"
    return f"postgresql://{user}:{password}@{host}:{port}/{db_name}"


def get_api_url():
    host = os.environ.get("API_HOST", "localhost")
    port = 5005 if host == "localhost" else 80
    return f"http://{host}:{port}"
```

<1> We use functions for getting the current config, rather than constants
    available at import time, because that allows client code to modify
    `os.environ` if it needs to.

<2> _config.py_ also defines some default settings, designed to work when
    running the code from the developer's local machine.footnote:[
    This gives us a local development setup that "just works" (as much as possible).
    You may prefer to fail hard on missing environment variables instead, particularly
    if any of the defaults would be insecure in production.]

An elegant Python package called
https://github.com/hynek/environ-config[_environ-config_] is worth looking
at if you get tired of hand-rolling your own environment-based config functions.

> [!TIP]
> Don't let this config module become a dumping ground that is full of things only vaguely related to config and that is then imported all over the place.
    Keep things immutable and modify them only via environment variables.
    If you decide to use a bootstrap script,
    you can make it the only place (other than tests) that config is imported to.

### Docker-Compose and Containers Config

We use a lightweight Docker container orchestration tool called _docker-compose_.
It's main configuration is via a YAML file (sigh):footnote:[Harry is a bit YAML-weary.
It's _everywhere_, and yet he can never remember the syntax or how it's supposed
to indent.]


.docker-compose config file (docker-compose.yml)
```yaml
version: "3"
services:

  app:  #<1>
    build:
      context: .
      dockerfile: Dockerfile
    depends_on:
      - postgres
    environment:  #<3>
      - DB_HOST=postgres  <4>
      - DB_PASSWORD=abc123
      - API_HOST=app
      - PYTHONDONTWRITEBYTECODE=1  #<5>
    volumes:  #<6>
      - ./src:/src
      - ./tests:/tests
    ports:
      - "5005:80"  <7>


  postgres:
    image: postgres:9.6  #<2>
    environment:
      - POSTGRES_USER=allocation
      - POSTGRES_PASSWORD=abc123
    ports:
      - "54321:5432"
```

<1> In the _docker-compose_ file, we define the different _services_
    (containers) that we need for our app. Usually one main image
    contains all our code, and we can use it to run our API, our tests,
    or any other service that needs access to the domain model.

<2> You'll probably have other infrastructure services, including a database.
    In production you might not use containers for this; you might have a cloud
    provider instead, but _docker-compose_ gives us a way of producing a
    similar service for dev or CI.

<3> The `environment` stanza lets you set the environment variables for your
    containers, the hostnames and ports as seen from inside the Docker cluster.
    If you have enough containers that information starts to be duplicated in
    these sections, you can use `environment_file` instead. We usually call
    ours _container.env_.

<4> Inside a cluster, _docker-compose_ sets up networking such that containers are
    available to each other via hostnames named after their service name.

<5> Pro tip: if you're mounting volumes to share source folders between your
    local dev machine and the container, the `PYTHONDONTWRITEBYTECODE` environment variable
    tells Python to not write _.pyc_ files, and that will save you from
    having millions of root-owned files sprinkled all over your local filesystem,
    being all annoying to delete and causing weird Python compiler errors besides.

<6> Mounting our source and test code as `volumes` means we don't need to rebuild
    our containers every time we make a code change.

<7> The `ports` section allows us to expose the ports from inside the containers
    to the outside worldfootnote:[On a CI server, you may not be able to expose
    arbitrary ports reliably, but it's only a convenience for local dev. You
    can find ways of making these port mappings optional (e.g., with
    _docker-compose.override.yml_).]—these correspond to the default ports we set
    in _config.py_.

> [!NOTE]
> Inside Docker, other containers are available through hostnames named after
    their service name. Outside Docker, they are available on `localhost`, at the
    port defined in the `ports` section.


### Installing Your Source as a Package

All our application code (everything except tests, really) lives inside an
_src_ folder:

.The src folder
```text
├── src
│   ├── allocation  #<1>
│   │   ├── config.py
│   │   └── ...
│   └── setup.py  <2>
```

<1> Subfolders define top-level module names. You can have multiple if you like.

<2> And _setup.py_ is the file you need to make it pip-installable, shown next.

.pip-installable modules in three lines (src/setup.py)
```python
from setuptools import setup

setup(
    name="allocation", version="0.1", packages=["allocation"],
)
```

That's all you need. `packages=` specifies the names of subfolders that you
want to install as top-level modules. The `name` entry is just cosmetic, but
it's required. For a package that's never actually going to hit PyPI, it'll
do fine.footnote:[For more _setup.py_ tips, see
https://oreil.ly/KMWDz[this article on packaging] by Hynek.]


### Dockerfile

Dockerfiles are going to be very project-specific, but here are a few key stages
you'll expect to see:

.Our Dockerfile (Dockerfile)
```dockerfile
FROM python:3.9-slim-buster

<1>
# RUN apt install gcc libpq (no longer needed bc we use psycopg2-binary)

<2>
COPY requirements.txt /tmp/
RUN pip install -r /tmp/requirements.txt

<3>
RUN mkdir -p /src
COPY src/ /src/
RUN pip install -e /src
COPY tests/ /tests/

<4>
WORKDIR /src
ENV FLASK_APP=allocation/entrypoints/flask_app.py FLASK_DEBUG=1 PYTHONUNBUFFERED=1
CMD flask run --host=0.0.0.0 --port=80
```

<1> Installing system-level dependencies
<2> Installing our Python dependencies (you may want to split out your dev from
    prod dependencies; we haven't here, for simplicity)
<3> Copying and installing our source
<4> Optionally configuring a default startup command (you'll probably override
    this a lot from the command line)

> [!TIP]
> One thing to note is that we install things in the order of how frequently they
    are likely to change. This allows us to maximize Docker build cache reuse. I
    can't tell you how much pain and frustration underlies this lesson. For this
    and many more Python Dockerfile improvement tips, check out
    https://pythonspeed.com/docker["Production-Ready Docker Packaging"].

### Tests

Our tests are kept alongside everything else, as shown here:

.Tests folder tree
```text
└── tests
    ├── conftest.py
    ├── e2e
    │   └── test_api.py
    ├── integration
    │   ├── test_orm.py
    │   └── test_repository.py
    ├── pytest.ini
    └── unit
        ├── test_allocate.py
        ├── test_batches.py
        └── test_services.py
```

Nothing particularly clever here, just some separation of different test types
that you're likely to want to run separately, and some files for common fixtures,
config, and so on.

There's no _src_ folder or _setup.py_ in the test folders because we usually
haven't needed to make tests pip-installable, but if you have difficulties with
import paths, you might find it helps.


### Wrap-Up

These are our basic building blocks:

* Source code in an _src_ folder, pip-installable using _setup.py_
* Some Docker config for spinning up a local cluster that mirrors production as far as possible
* Configuration via environment variables, centralized in a Python file called _config.py_, with defaults allowing things to run _outside_ containers
* A Makefile for useful command-line, um, commands

We doubt that anyone will end up with _exactly_ the same solutions we did, but we hope you
find some inspiration here.
