python-packaging | Tutorial on how to structure Python packages | Build Tool library

User configuration should usually be kept in the os-specific locations for application configs. The usual way to handle it as a package author is to

• write a default config file and put it into the package, e.g. at src/my_package/default.ini - this is better than writing the config as code, as your main.py probably does, because you can re-use your parsing step, can copy the default config to get a valid user config, and just have to maintain a single file type
• copy the default config into a user-specific directory, use appdirs to find out which one is correct on each os
  • you can either ask your users if it's ok to create a new config at e.g. /home/user/.configs/my_app and do it after confirmation, or just do it and tell them where to find it in case they want to customize something
• on application start, parse the default config first, then parse the user config and overwrite differing settings
• if you also allow configuration via environment variables, those will be parsed afterwards and should overwrite both default- and user-config

Just to have a starting point, my config-parsing usually looks somewhat like this:

config.py, somewhere in your package

The libcurl-devel & openssl-devel packages are missing, use one of the lambci/lambda Docker images to build and make sure all the dependencies are installed.

Change your serverless.yml to:

I think that relocatable environments depend on the installed packages. They should be implemented with relative paths and avoiding hardcoded paths. All the paths that are used in the source code of the package should be inside the own package. So if you install well done package you won´t have any problem to relocate the environment in other folder or computer.

As you will need to add all the folders inside the package you will need to modify the arguments of the setup. Add these two parameters in order to add folders to the final package. If you don´t do this the folders won´t copied to the site-packages folder within the environment (the final destination when you install the package with pip):

As the links above do not mentionned, we need to this extra line in setup.py :

The guide you are following is 5 years old and a little vague. You could setup your cli-tool like so:

Python's doc also says:

The Distutils install command is designed to make installing module distributions to an alternate location simple and painless. The basic idea is that you supply a base directory for the installation, and the install command picks a set of directories (called an installation scheme) under this base directory in which to install files. The details differ across platforms, so read whichever of the following sections applies to you.

Note that the various alternate installation schemes are mutually exclusive: you can pass --user, or --home, or --prefix and --exec-prefix, or --install-base and --install-platbase, but you can’t mix from these groups.

See it here

It seems like you missed step about adding serverless-python-requirements plugin to serverless.yml.

Try adding it as mentioned blog:

As you've mentioned from the error you received it looks like your zipped package is too large. You received that other error because you have a module requirement in your script to use tensorflow.

Keep in mind the AWS Lambda Limits has a 50MB deployment package size limit. The Tensorflow package by itself is close to 50MB, so adding the Numpy package will take it well over the limit.

Have a look at this blog which does some investigation into the package limit sizing in AWS Lambda

https://hackernoon.com/exploring-the-aws-lambda-deployment-limits-9a8384b0bec3

These are many broad questions. It is difficult to give an exhaustive answer and to keep opinions aside...

When it comes strictly to packaging Python projects, I believe pip's "requirements" files are not commonly useful nor used (there are counter examples of course, but I would argue that they are not a good idea for packaging in general). But in some other use cases, I believe they are very helpful.

If setuptools is used for packaging (there are perfectly valid alternatives), then install_requires is the way to go to declare the direct dependencies of the project. Unless there is no other way around, then I would recommend placing the whole setuptools configuration (this includes install_requires) in the setup.cfg file instead of as arguments to setuptools.setup(). The result would be exactly the same in both cases but (as it is now commonly accepted) for configuration, a static declarative file is a much more natural fit than executable code. As for the order of precedence, I believe a setuptools.setup() argument would override the same configuration item read from setup.cfg.

Would e.g. pip first execute setup.py and then check the requirements.txt afterwards?

These are two different things. The requirements file is more or less a list of projects that one wants to install. Each of these projects might be packaged (setuptools or other) and distributed (source or pre-built wheel) in different ways, some will require the execution of a setup.py some won't. But again, when packaging (and distributing) a project there is usually no need to bother with a requirements file (never when the project is a library, and not at first when the project is an application, see Donald Stufft's article "setup.py vs requirements.txt").

I would rather not go into details regarding the question about the combination of Cython and setuptools. It would be a good additional question. These two links might help get you started though:

• https://setuptools.readthedocs.io/en/latest/setuptools.html#distributing-extensions-compiled-with-cython
• https://medium.com/@grassfedcode/pep-517-and-518-in-plain-english-47208ca8b7a6