Development project teams generally have a starter template which avoids a lot of
these steps; they are provided here just as an illustration of what's happening.
We are going to use a few Python packages for Django and various add-ons. This mostly-complete list of requirements
is here, but, in practice, you would build up this list over time as you develop your project.
Django is the core Django framework.
Django Debug Toolbar adds an in-browser debugger for the Django UI.
Django REST Framework (DRF) makes it easy to write RESTful APIs.
Django REST Framework JSON API (DJA) extends DRF to use the
{json:api} format.
Django OAuth Toolkit (DOT) adds an OAuth 2.0-based security layer.
Django REST condition allows for boolean composition of DRF view permissions.
Django Filter for filtering results using the {json:api} filter query parameter.
PyYAML for YAML file utilities.
tox for automated unit tests, etc.
tox-pip-extensions makes tox work better with pip.
For the complete list of required packages and any version constraints, see requirements.txt.
Following is an example of "manually" adding the packages, one at a time, but you
would more likely use pip install -r requirements.txt as shown below.
To make sure your code is working against a consistent known set of package versions, it's a good
idea to "pin" or "freeze" your installed package versions. You will want to upgrade these from time to time though.
Next time you or someone else works on a clone of your project, all the preceding steps can be replaced by
pip install -r requirements.txt.
See below for an example of a more sophisticated requirements.txt that uses
specific version ranges and pre-released package versions to work around some bugs or use new features.
Make sure git ignores irrelevant (non-source) files¶
We want to ignore our virtualenv directory, and various output files created by IDEs, editors, tox,
compiled python and so on.
(env)django-jsonapi-training$ django-admin startproject training .(env)django-jsonapi-training$ django-admin startapp myapp(env)django-jsonapi-training$ ./manage.py migrateOperations to perform: Apply all migrations: admin, auth, contenttypes, sessionsRunning migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying sessions.0001_initial... OK(env)django-jsonapi-training$ ./manage.py createsuperuserUsername (leave blank to use 'ac45'): adminEmail address: Password: admin123 Password (again): admin123 This password is too common.Bypass password validation and create user anyway? [y/N]: ySuperuser created successfully.
Let's look at what's been created. We'll ignore the env directory as that's where the virtualenv stuff
lives, including all the Python packages in env/lib/python3.6/site-packages/.
(env)django-jsonapi-training$ lsdb.sqlite3 env/ manage.py* myapp/ requirements.txt training/(env)django-jsonapi-training$ tree myapp trainingmyapp├── __init__.py├── admin.py├── apps.py├── migrations│ └── __init__.py├── models.py├── tests.py└── views.pytraining├── __init__.py├── __pycache__│ ├── __init__.cpython-36.pyc│ ├── settings.cpython-36.pyc│ └── urls.cpython-36.pyc├── settings.py├── urls.py└── wsgi.py2 directories, 14 files(env)django-jsonapi-training$ git statusOn branch masterNo commits yetUntracked files: (use "git add <file>..." to include in what will be committed) .gitignore manage.py myapp/ requirements.txt training/nothing added to commit but untracked files present (use "git add" to track)(env)django-jsonapi-training$ git add .(env)django-jsonapi-training$ git statusOn branch masterNo commits yetChanges to be committed: (use "git rm --cached <file>..." to unstage) new file: .gitignore new file: manage.py new file: myapp/__init__.py new file: myapp/admin.py new file: myapp/apps.py new file: myapp/migrations/__init__.py new file: myapp/models.py new file: myapp/tests.py new file: myapp/views.py new file: requirements.txt new file: training/__init__.py new file: training/settings.py new file: training/urls.py new file: training/wsgi.py(env)django-jsonapi-training$ git commit -m "initial project"[master (root-commit) f6c154d] initial project 14 files changed, 221 insertions(+) create mode 100644 .gitignore create mode 100755 manage.py create mode 100644 myapp/__init__.py create mode 100644 myapp/admin.py create mode 100644 myapp/apps.py create mode 100644 myapp/migrations/__init__.py create mode 100644 myapp/models.py create mode 100644 myapp/tests.py create mode 100644 myapp/views.py create mode 100644 requirements.txt create mode 100644 training/__init__.py create mode 100644 training/settings.py create mode 100644 training/urls.py create mode 100644 training/wsgi.py(env)django-jsonapi-training$ git logcommit f6c154d15771c01e3034a75024b308d0db36ae8d (HEAD -> master)Author: Alan Crosswell <alan@columbia.edu>Date: Fri Oct 26 16:36:49 2018 -0400 initial project(env)django-jsonapi-training$ git tag initial
You can now use the above commit as a template to start future projects if you like.
Browse the source code for this project or clone it. Most of the code is reproduced below as well,
but is likely not completely up to date. Here's a summary of cloning, assuming you've already
setup your git SSH keys:
123
src$ git clone git@github.com:columbia-it/django-jsonapi-training.gitsrc$ cd django-jsonapi-trainingdjango-jsonapi-training$ git checkout initial
If you want to follow along, you can uses the various git tags to check out pieces of the project.
We'll indicate them like this:
GIT TAG: initial
Edit Settings to add DRF, DJA, OAuth, Debug, etc.¶
GIT TAG: settings
An initial version of training/settings.py is created by
django-admin startproject and django-admin startapp. It's full of comments suggesting changes.
We will make the following additions to the default settings.py:
Weaken security (allowed hosts) for CORS.
Identify when to show the debug toolkit (internal IPs)
Add installed apps
Add middleware
Configure admin view permissions
Configure DRF and DJA
Configure DOT
Enable debug logging so we can see what's happening
Parametrize various credentials and options using environment variables
Configure an optional external Microsoft SQLServer database
You can just take a look at the latest version of settings.py. Following is the
diff between the initial boilerplate code and our edits.
Generally, you need to have an application architecture sketched out in advance. This includes a data model.
You can start by sketching your entity-relationship diagrams on a whiteboard or use a modeling tool to generate a
UML
diagram. Or, you can just make Django models
and then pretend you designed them first using the django-extensions package. Your manager will never known;-)
Here's our model with the CommonModel off to the side to make things more readable:
django-admin startapp created a starter myapp/models.py. Now add some actual model definitions to it.
These are just like in "vanilla" Django.
We're going to:
1. Make a CommonModel abstract class that adds some common fields that we want all our models to include.
These are things like various dates, modifying users, etc. We also choose to make our id a UUID4
as recommended in the {json:api} spec.
2. Create a Course "parent" model and a CourseTerm "child model" that references each Course instance via
a ForeignKey.
3. Each model has a default parameter to order by. These will come in handy when we get to pagination and want
consistent paginated results.
importuuidfromdjango.dbimportmodelsclassCommonModel(models.Model):""" Abstract model with common fields for all "real" Models: - id: globally unique UUID version 4 - effective dates - last modified dates """id=models.UUIDField(primary_key=True,default=uuid.uuid4,editable=False)effective_start_date=models.DateField(default=None,blank=True,null=True)effective_end_date=models.DateField(default=None,blank=True,null=True)last_mod_user_name=models.CharField(default=None,null=True,max_length=80)last_mod_date=models.DateField(auto_now=True)classMeta:abstract=TrueclassCourse(CommonModel):""" A course of instruction. e.g. COMSW1002 Computing in Context """school_bulletin_prefix_code=models.CharField(max_length=10)suffix_two=models.CharField(max_length=2)subject_area_code=models.CharField(max_length=10)course_number=models.CharField(max_length=10)course_identifier=models.CharField(max_length=10,unique=True)course_name=models.CharField(max_length=80)course_description=models.TextField()classMeta:ordering=["course_number"]def__str__(self):return'%s,%s,%s,%s'%(self.id,self.course_number,self.course_identifier,self.course_name)classCourseTerm(CommonModel):""" A specific course term (year+semester) instance. e.g. 20183COMSW1002 """term_identifier=models.TextField(max_length=10)audit_permitted_code=models.PositiveIntegerField(blank=True,default=0)exam_credit_flag=models.BooleanField(default=True)course=models.ForeignKey('myapp.Course',related_name='course_terms',on_delete=models.CASCADE,null=True,default=None)classMeta:ordering=["term_identifier"]def__str__(self):return'%s,%s,%s'%(self.id,self.term_identifier,self.course.course_identifier)
Serializers render the Models in the "wire" format, which is JSON, and specifically {json:api},
so we import our serializers from rest_framework_json_api.serializers.
We will:
1. Use a HyperlinkedModelSerializer which gives us the HATEOAS links.
2. For each Model, choose which model fields to serialize.
3. Define the ResourceRelatedField linkage between the models and the {json:api} representation.
This is where the {json:api} relationships and related attributes get created.
4. Add included_serializers which are needed to serialize the included compound document data.
fromrest_framework_json_api.relationsimportResourceRelatedFieldfromrest_framework_json_api.serializersimportHyperlinkedModelSerializerfrommyapp.modelsimportCourse,CourseTermclassCourseSerializer(HyperlinkedModelSerializer):""" (de-)serialize the Course. """classMeta:model=Coursefields=('url','school_bulletin_prefix_code','suffix_two','subject_area_code','course_number','course_identifier','course_name','course_description','effective_start_date','effective_end_date','last_mod_user_name','last_mod_date','course_terms')course_terms=ResourceRelatedField(model=CourseTerm,many=True,read_only=False,allow_null=True,required=False,queryset=CourseTerm.objects.all(),self_link_view_name='course-relationships',related_link_view_name='course-related',)# {json:api} 'included' support (also used for `related_serializers` for DJA 2.6.0)included_serializers={'course_terms':'myapp.serializers.CourseTermSerializer',}# Uncomment this and the course_terms will be included by default,# otherwise '?include=course_terms' must be added to the URL.# class JSONAPIMeta:# included_resources = ['course_terms']classCourseTermSerializer(HyperlinkedModelSerializer):classMeta:model=CourseTermfields=('url','term_identifier','audit_permitted_code','exam_credit_flag','effective_start_date','effective_end_date','last_mod_user_name','last_mod_date','course')course=ResourceRelatedField(model=Course,many=False,# this breaks new 2.6.0 related support. Only works when True.read_only=False,allow_null=True,required=False,queryset=Course.objects.all(),self_link_view_name='course_term-relationships',related_link_view_name='course_term-related',)# {json:api} 'included' supportincluded_serializers={'course':'myapp.serializers.CourseSerializer',}
Note the Django style of forward-referencing classes that have not yet been defined: In the included_serializers
you can either provide a reference to the serializer (e.g. {'course': CourseSerializer}) or a string containing
the full Python path name of the serializer ({'course': 'myapp.serializers.CourseSerializer'}). In a case like
the one shown here, where the first serializer references the second and vice-versa, there will always be one
that is referenced before being defined, so we just consistently use the string reference style. (We also saw this
in settings.py for INSTALLED_APPS and so on.)
I could have made life easier for myself by using fields = "__all__"" which tells the ModelSerializer to
just include all the fields of the model plus the additional fields defined in this class (and, for the
HyperlinkedModelSerializer, also the url field). Also, to keep things DRY (Don't Repeat Yourself) I've tried to
minimize repetive code:
diff --git a/myapp/serializers.py b/myapp/serializers.pyindex 32604cb..beb962c 100644--- a/myapp/serializers.py+++ b/myapp/serializers.py@@ -8,11 +8,24 @@ from myapp.models import Course, CourseTerm, Instructor, Person
class HyperlinkedModelSerializer(HyperlinkedModelSerializer):
"""
+ Common serializer class for all model serializers.
Extends :py:class:`.models.CommonModel` to set `last_mod_user_name` and `...date` from auth.user on a
POST/PATCH, not from the client app.
+ This silently *ignores* anything CREATEd or PATCHed for these fields.
"""
- #: these are read-only fields- read_only_fields = ('last_mod_user_name', 'last_mod_date')+ class Meta:+ """+ In order for this Meta inner class to be inherited by the various serializers,+ one must explicitly inherit it as in this example::++ class MySerializer(HyperlinkedModelSerializer):+ class Meta(HyperlinkedModelSerializer.Meta):+ model = MyModel+ """+ #: serialize all model fields unless otherwise overridden+ fields = "__all__"+ #: mark these fields as read-only+ read_only_fields = ('last_mod_user_name', 'last_mod_date')
def _last_mod(self, validated_data):
"""
@@ -40,18 +53,12 @@ class HyperlinkedModelSerializer(HyperlinkedModelSerializer):
class CourseSerializer(HyperlinkedModelSerializer):
"""
- (de-)serialize the Course.+ (de-)serialize the Course model.
"""
- class Meta:+ class Meta(HyperlinkedModelSerializer.Meta):
model = Course
- fields = (- 'url',- 'school_bulletin_prefix_code', 'suffix_two', 'subject_area_code',- 'course_number', 'course_identifier', 'course_name', 'course_description',- 'effective_start_date', 'effective_end_date',- 'last_mod_user_name', 'last_mod_date',- 'course_terms')+ #: a course has zero or more course_term instances
course_terms = ResourceRelatedField(
model=CourseTerm,
many=True,
@@ -63,7 +70,8 @@ class CourseSerializer(HyperlinkedModelSerializer):
related_link_view_name='course-related',
)
- #: json api 'included' support (also used for `related_serializers` for DJA 2.6.0)+ #: `{json:api} compound document <https://jsonapi.org/format/#document-compound-documents>`_+ #: (also used for `related_serializers` for DJA 2.6.0)
included_serializers = {
'course_terms': 'myapp.serializers.CourseTermSerializer',
}
@@ -74,16 +82,13 @@ class CourseSerializer(HyperlinkedModelSerializer):
class CourseTermSerializer(HyperlinkedModelSerializer):
- class Meta:+ """+ (de-)serialize the CourseTerm model.+ """+ class Meta(HyperlinkedModelSerializer.Meta):
model = CourseTerm
- fields = (- 'url',- 'term_identifier', 'audit_permitted_code',- 'exam_credit_flag',- 'effective_start_date', 'effective_end_date',- 'last_mod_user_name', 'last_mod_date',- 'course', 'instructors')+ #: a course_term has zero or one parent courses
course = ResourceRelatedField(
model=Course,
many=False,
@@ -94,6 +99,7 @@ class CourseTermSerializer(HyperlinkedModelSerializer):
self_link_view_name='course_term-relationships',
related_link_view_name='course_term-related',
)
+ #: a course_term can have many instructors
instructors = ResourceRelatedField(
model=Instructor,
many=True,
@@ -105,7 +111,8 @@ class CourseTermSerializer(HyperlinkedModelSerializer):
related_link_view_name='course_term-related',
)
- #: json api 'included' support+ #: ``?include=course`` or ``?include=instructors``+ #: `{json:api} compound document <https://jsonapi.org/format/#document-compound-documents>`_
included_serializers = {
'course': 'myapp.serializers.CourseSerializer',
'instructors': 'myapp.serializers.InstructorSerializer',
@@ -113,10 +120,13 @@ class CourseTermSerializer(HyperlinkedModelSerializer):
class PersonSerializer(HyperlinkedModelSerializer):
- class Meta:+ """+ (de-)serialize the Person model.+ """+ class Meta(HyperlinkedModelSerializer.Meta):
model = Person
- fields = ('url', 'name', 'instructor')+ #: a person is an instructor
instructor = ResourceRelatedField(
model=Instructor,
many=False,
@@ -128,16 +138,21 @@ class PersonSerializer(HyperlinkedModelSerializer):
related_link_view_name='person-related',
)
+ #: `{json:api} compound document <https://jsonapi.org/format/#document-compound-documents>`_
included_serializers = {
'instructor': 'myapp.serializers.InstructorSerializer',
}
class InstructorSerializer(HyperlinkedModelSerializer):
- class Meta:+ """+ (de-)serialize the Instructor model.+ """+ class Meta(HyperlinkedModelSerializer.Meta):
model = Instructor
- fields = ('person', 'course_terms', 'url')+ fields = "__all__"+ #: an instructor teaches zero or more course instances
course_terms = ResourceRelatedField(
model=CourseTerm,
many=True,
@@ -149,6 +164,7 @@ class InstructorSerializer(HyperlinkedModelSerializer):
related_link_view_name='instructor-related',
)
+ #: an instructor is a person
person = ResourceRelatedField(
model=Person,
many=False,
@@ -160,6 +176,7 @@ class InstructorSerializer(HyperlinkedModelSerializer):
related_link_view_name='instructor-related'
)
+ #: `{json:api} compound document <https://jsonapi.org/format/#document-compound-documents>`_
included_serializers = {
'course_terms': 'myapp.serializers.CourseTermSerializer',
'person': 'myapp.serializers.PersonSerializer',
Note that I've followed a somewhat common pattern of extending a class using the same name as the
base class (HyperlinkedModelSerialzer). This can be somewhat confusing at first glance but also
makes it easy to add functionality without changing a lot of source code.
A view function is defined for each HTTP endpoint in the app. DRF uses Class-based views (CBV) in which
the ViewSet class has an as_view() function that returns a view function. The HTTP endpoints are
defined in the urlpatterns list in urls.py and reference the CBV's in views.py.
See training/urls.py.
Let's start with the URL routing:
1. Version our API by prepending /v1 in front of our resources.
1. Redirect from / to /v1 (the current version).
1. Use the DRF DefaultRouter to generate "the usual" URL routes to our ViewSets
(CourseViewSet and CourseTermViewSet)
1. Because DJA doesn't (yet) have a DefaultRouter, we then have to explicitly add
routes for each resource's relationships and related attributes.
We'll explain the kwargs that are set with <arg> in each path() when we get to the views.
"""training URL ConfigurationThe `urlpatterns` list routes URLs to views. For more information please see: https://docs.djangoproject.com/en/2.1/topics/http/urls/Examples:Function views 1. Add an import: from my_app import views 2. Add a URL to urlpatterns: path('', views.home, name='home')Class-based views 1. Add an import: from other_app.views import Home 2. Add a URL to urlpatterns: path('', Home.as_view(), name='home')Including another URLconf 1. Import the include() function: from django.urls import include, path 2. Add a URL to urlpatterns: path('blog/', include('blog.urls'))"""fromdjango.confimportsettingsfromdjango.contribimportadminfromdjango.contrib.authimportviewsasauth_viewsfromdjango.contrib.staticfiles.urlsimportstaticfiles_urlpatternsfromdjango.urlsimportinclude,pathfromdjango.views.generic.baseimportRedirectViewfromrest_frameworkimportroutersfrommyappimportviewsAPI_TITLE='Demo API'admin.autodiscover()router=routers.DefaultRouter()router.register(r'courses',views.CourseViewSet)router.register(r'course_terms',views.CourseTermViewSet)urlpatterns=[path('',RedirectView.as_view(url='/v1',permanent=False)),path('v1/',include(router.urls)),# TODO: Is there a Router than can generate these for me? If not, create one.# course relationships:path('v1/courses/<pk>/relationships/<related_field>/',views.CourseRelationshipView.as_view(),name='course-relationships'),# use new `retrieve_related` functionality in DJA 2.6.0 which hangs all the related serializers off the parent.# (we only have one relationship in the current model so this doesn't really demonstrate the power).path('v1/courses/<pk>/<related_field>/',views.CourseViewSet.as_view({'get':'retrieve_related'}),name='course-related'),# course_terms relationshipspath('v1/course_terms/<pk>/relationships/<related_field>/',views.CourseTermRelationshipView.as_view(),name='course_term-relationships'),# use new `retrieve_related` functionality in DJA 2.6.0:path('v1/course_terms/<pk>/<related_field>/',views.CourseTermViewSet.as_view({'get':'retrieve_related'}),# a toOne relationshipname='course_term-related'),# browseable API and admin stuff. TODO: Consider leaving out except when debugging.path('api-auth/',include('rest_framework.urls',namespace='rest_framework')),path('admin/',admin.site.urls),path('accounts/login/',auth_views.LoginView.as_view()),path('accounts/logout/',auth_views.LogoutView.as_view(),{'next_page':'/'},name='logout'),]urlpatterns+=staticfiles_urlpatterns()ifsettings.DEBUG:importdebug_toolbarurlpatterns=[path('__debug__/',include(debug_toolbar.urls)),]+urlpatterns
See myapp/views.py for the views.
The views is where a lot of the work happens regarding authentication and authorization. We're going
to start simple and leave all that out initially so we can focus on our ViewSets.
fromrest_framework_json_api.viewsimportModelViewSet,RelationshipViewfrommyapp.modelsimportCourse,CourseTermfrommyapp.serializersimportCourseSerializer,CourseTermSerializerclassCourseViewSet(ModelViewSet):""" API endpoint that allows course to be viewed or edited. """queryset=Course.objects.all()serializer_class=CourseSerializerclassCourseTermViewSet(ModelViewSet):""" API endpoint that allows CourseTerm to be viewed or edited. """queryset=CourseTerm.objects.all()serializer_class=CourseTermSerializerclassCourseRelationshipView(RelationshipView):""" view for relationships.course """queryset=Course.objectsself_link_view_name='course-relationships'classCourseTermRelationshipView(RelationshipView):""" view for relationships.course_terms """queryset=CourseTerm.objectsself_link_view_name='course_term-relationships'
Now we'll make sure our database is in sync with the newly-defined app models and then apply migrations
that include those for Django core and various add-on packages as well as myapp.
(Because we've configured debug logging for the database in settings.py,
let's override DEBUG for the time being to reduce the noise.
Recall that the DJANGO_DEBUG environment variable is used to customize settings.DEBUG.)
(env)django-jsonapi-training$ export DJANGO_DEBUG=false(env)django-jsonapi-training$ ./manage.py makemigrationsMigrations for 'myapp': myapp/migrations/0001_initial.py - Create model Course - Create model CourseTerm(env)django-jsonapi-training$ ./manage.py migrateOperations to perform: Apply all migrations: admin, auth, contenttypes, myapp, oauth2_provider, sessionsRunning migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying myapp.0001_initial... OK Applying oauth2_provider.0001_initial... OK Applying oauth2_provider.0002_08_updates... OK Applying oauth2_provider.0003_auto_20160316_1503... OK Applying oauth2_provider.0004_auto_20160525_1623... OK Applying oauth2_provider.0005_auto_20170514_1141... OK Applying oauth2_provider.0006_auto_20171214_2232... OK Applying sessions.0001_initial... OK
Anytime things are feeling confusing, just remove the sqlite3 database and re-migrate.
A small subset of this data is available in a
test fixture
in myapp/fixtures/testcases.yaml which can be
conveniently loaded into the database as follows:
12
(env)django-jsonapi-training$ ./manage.py loaddata myapp/fixtures/testcases.yamlInstalled 28 object(s) from 1 fixture(s)
To make life easy, we started the example above with no
authentication (identifying clients)
or authorization (permission to GET, POST, PATCH, DELETE resources).
Let's add them in to views.py:
The client app can have a live person using it, using the Authorization Code or Implicit grant type, or,
it can be a program, using the Client Credentials grant.
These are indicated by the use of a Columbia-specific OAuth 2.0 scope configuration:
- auth-columbia indicates that a Shibboleth/CAS login is required.
- auth-none indicates that there is no user login.
OAuth 2.0 clients are (carefully) configured such that the above scopes are only grantable
in the "right" cases. (e.g. auth-none scope is not generally available unless the client app is a trusted
program that has properly secured its OAuth 2 client credentials.)
In addition to OAuth 2.0, to make life easy for testing and perhaps using the DRF Browseable API
(which is not meant to be used for production clients), we'll add options for
HTTP Basic Auth and Session cookies.
Additional OAuth 2.0 scopes are used to authorize those clients (CRUD: create, read, update, delete)
as well as app-specific scopes such as demo-netphone-admin. (TODO: change this to a meaningful name!)
N.B. These are all Columbia-specific scope configurations.
As an alternative to OAuth 2.0 scopes for permissions, using DRF's bitwise operators,
we'll also take advantage of the Django authorization framework's
DjangoModelPermissions
for known users. The key thing we do here is disallow blanket GET by just anyone, which is the Django default.
Again, for this sample app, DjangoModelPermissions don't apply; it's all about OAuth 2.0 scopes. A real app
will likely do the ors and ands differently.
Lot's more can be done with Django's permission system such as adding object and/or field-level permissions.
{"links":{"first":"http://127.0.0.1:8000/v1/courses/?fields%5Bcourses%5D=course_number%2Ccourse_name&page%5Bnumber%5D=1&page%5Bsize%5D=3&sort=-course_number","last":"http://127.0.0.1:8000/v1/courses/?fields%5Bcourses%5D=course_number%2Ccourse_name&page%5Bnumber%5D=1491&page%5Bsize%5D=3&sort=-course_number","next":"http://127.0.0.1:8000/v1/courses/?fields%5Bcourses%5D=course_number%2Ccourse_name&page%5Bnumber%5D=43&page%5Bsize%5D=3&sort=-course_number","prev":"http://127.0.0.1:8000/v1/courses/?fields%5Bcourses%5D=course_number%2Ccourse_name&page%5Bnumber%5D=41&page%5Bsize%5D=3&sort=-course_number"},"data":[{"type":"courses","id":"ad326e70-bc7f-4068-a67e-75e0ca6cbb50","attributes":{"course_number":"93635","course_name":"EVID & POLITICS OF HLTH POL"},"links":{"self":"http://127.0.0.1:8000/v1/courses/ad326e70-bc7f-4068-a67e-75e0ca6cbb50/"}},{"type":"courses","id":"09f60887-8e2b-494d-8ff8-d83077488d0e","attributes":{"course_number":"93634","course_name":"PHILOSOPHY OF BIOETHICS"},"links":{"self":"http://127.0.0.1:8000/v1/courses/09f60887-8e2b-494d-8ff8-d83077488d0e/"}},{"type":"courses","id":"a903c98a-bd4b-4da4-8470-3058b7dcc94d","attributes":{"course_number":"93633","course_name":"BASKETBALL ANALYTICS"},"links":{"self":"http://127.0.0.1:8000/v1/courses/a903c98a-bd4b-4da4-8470-3058b7dcc94d/"}}],"meta":{"pagination":{"page":42,"pages":1491,"count":4473}}}
See settings.py for where the REST_FRAMEWORK default classes are configured.
You can also add these classes on a per-view basis using, for example, the .pagination_class attribute.
{"links":{"first":"http://127.0.0.1:8000/v1/courses/?fields%5Bcourses%5D=course_name%2Ccourse_description&filter%5Bsearch%5D=data+analytics&page%5Bnumber%5D=1","last":"http://127.0.0.1:8000/v1/courses/?fields%5Bcourses%5D=course_name%2Ccourse_description&filter%5Bsearch%5D=data+analytics&page%5Bnumber%5D=1","next":null,"prev":null},"data":[{"type":"courses","id":"0e3f05e6-ab39-4e4c-82d3-10ba329e4cdb","attributes":{"course_name":"TOPICS-INFORMATION PROCESSING","course_description":"TPC: ADV BIG DATA ANALYTICS"},"links":{"self":"http://127.0.0.1:8000/v1/courses/0e3f05e6-ab39-4e4c-82d3-10ba329e4cdb/"}},{"type":"courses","id":"4fa73679-aeef-482d-97b2-fab1e9e92cae","attributes":{"course_name":"IoT - SYS &PHY DATA ANALYTICS","course_description":"IoT - SYS & PHY DATA ANALYSIS"},"links":{"self":"http://127.0.0.1:8000/v1/courses/4fa73679-aeef-482d-97b2-fab1e9e92cae/"}},{"type":"courses","id":"4fd59f80-4359-4a94-a610-30284a0b5e55","attributes":{"course_name":"DATA ANALYTICS/METRICS IN NONPROF SECTOR","course_description":"DATA ANL/MTRC IN NONPROF"},"links":{"self":"http://127.0.0.1:8000/v1/courses/4fd59f80-4359-4a94-a610-30284a0b5e55/"}}],"meta":{"pagination":{"page":1,"pages":1,"count":3}}}
In our example, we configure filterset_fields with a variety of relational operations. Note that
some of these perform related field path searches, for example: course_terms__term_identifier.
This is configured using the standard Django double-underscore notation but can also use {json:api} dotted
notation: course_terms.term_identifier.
Note that some conditions are not easily represented, including NOT and empty string (as contrasted with null).
Hint: try a regex (school_bulletin_prefix_code not starting with "M"):
Say we want to prevent the client from updating who the last_mod_user_name was or when the last_mod_date
happened. This example silently overrides the serializer create and update methods to use the authenticated
user and current date when updating the underlying Model. Let's mark those fields read-only as well.
diff --git a/myapp/serializers.py b/myapp/serializers.pyindex 4075d9c..cabd404 100644--- a/myapp/serializers.py+++ b/myapp/serializers.py@@ -6,6 +6,36 @@ from rest_framework_json_api.serializers import HyperlinkedModelSerializer
from myapp.models import Course, CourseTerm
+class HyperlinkedModelSerializer(HyperlinkedModelSerializer):+ """+ .models.CommonModel.last_mod_user_name/date should come from auth.user on a POST/PATCH, not from the client app.+ """+ read_only_fields = ('last_mod_user_name', 'last_mod_date')++ def _last_mod(self, validated_data):+ """+ override any last_mod_user_name or date with current auth user and current date.+ """+ # N.B. if OAuth2 Client Credentials is used, there is no user *and* the `client_id` is not+ # currently properly tracked for an external AS: https://github.com/jazzband/django-oauth-toolkit/issues/664+ validated_data['last_mod_user_name'] = str(self.context['request'].user)+ validated_data['last_mod_date'] = datetime.now().date()++ def create(self, validated_data):+ """+ extended ModelSerializer to set last_mod_user/date+ """+ self._last_mod(validated_data)+ return super(HyperlinkedModelSerializer, self).create(validated_data)++ def update(self, instance, validated_data):+ """+ extended ModelSerializer to set last_mod_user/date+ """+ self._last_mod(validated_data)+ return super(HyperlinkedModelSerializer, self).update(instance, validated_data)
The serializer is probably not the right place for "business logic" in general as it allows Model
manipulation to bypass that logic. It should probably
happen in the Model as that
is the closest layer to the data and we want to always enforce the business rules.
The main reason the preceding code is implemented in the serializer is that the Model does not have the
current request context directly available, so it is hard to identify who the user is. In fact, if the Model
is being manipulated by code that runs outside the context of an HTTP request, there is no user! This is an argument
for leaving the user-specific code in the serializer.
If you need to do it in the Model, you'll want to use
thread local data
and grab the request.user in a Django Middleware
function and store it in thread local storage so it can be retrieved in the Model manager. Also, take a look
at the Django Signals
documentation for some alternative ideas.
Let's start with the basics and make sure our Models make sense. Django has a nice testing framework that,
by default, makes a new in-memory sqlite database each time a test suite is run. This guarantees that the
test data is always in a known state.
See myapp/tests/test_models.py
We'll create a few Course and CourseTerm objects and do some trivial tests to make sure they work.
fromdatetimeimportdate,datetimefromdjango.db.utilsimportIntegrityErrorfromdjango.testimportTestCasefrommyapp.modelsimportCourse,CourseTermclassCourseTestCase(TestCase):defsetUp(self):c1=Course.objects.create(school_bulletin_prefix_code='123',suffix_two='11',subject_area_code='A101',course_number='123',course_identifier='ABC123',course_name='Corp. Finance',course_description='bla bla bla 123',last_mod_user_name='Fred Gary')c1.save()c2=Course.objects.create(school_bulletin_prefix_code='888',suffix_two='22',subject_area_code='A102',course_number='456',course_identifier='XYZ321',course_name='Data Science III',course_description='bla bla bla 456',last_mod_user_name='Tom Smith')c2.save()CourseTerm.objects.create(term_identifier='term_id_1',audit_permitted_code=123,exam_credit_flag=True,last_mod_user_name='Fred Gary',effective_start_date=datetime.now(),effective_end_date=date(2019,1,31),course=c1).save()CourseTerm.objects.create(term_identifier='term_id_2',audit_permitted_code=222,exam_credit_flag=False,last_mod_user_name='Tom Frank',effective_start_date=datetime.now(),effective_end_date=date(2019,2,28),course=c1).save()CourseTerm.objects.create(term_identifier='term_id_21',audit_permitted_code=321,exam_credit_flag=True,last_mod_user_name='Fred Gary',effective_start_date=datetime.now(),effective_end_date=date(2020,1,31),course=c2).save()CourseTerm.objects.create(term_identifier='term_id_22',audit_permitted_code=422,exam_credit_flag=False,last_mod_user_name='Tom Frank',last_mod_date=datetime.now(),effective_start_date=datetime.now(),effective_end_date=date(2020,12,28),course=c2).save()deftest_str(self):courses=Course.objects.all()c=courses.get(course_identifier='ABC123')self.assertEquals(c.course_identifier,'ABC123')deftest_date(self):courses=Course.objects.all()c=courses.get(course_identifier='ABC123')self.assertEquals(c.last_mod_date,date.today())deftest_num(self):cs=Course.objects.all()self.assertEquals(2,cs.count())terms=CourseTerm.objects.all()self.assertEquals(4,terms.count())# field look upts=CourseTerm.objects.filter(course__course_identifier='123ABC')self.assertEquals(0,ts.count())ts=CourseTerm.objects.filter(course__course_identifier='ABC123')self.assertEquals(2,ts.count())ts=CourseTerm.objects.filter(last_mod_user_name__startswith='Tom')self.assertEquals(2,ts.count())cs=Course.objects.filter(course_terms__last_mod_user_name__startswith='Tom')self.assertEquals(2,cs.count())deftest_dup_fail(self):c1=Course.objects.create(school_bulletin_prefix_code='123',suffix_two='11',subject_area_code='A101',course_number='123',course_identifier='ABC123',course_name='Corp. Finance',course_description='bla bla bla 123',last_mod_user_name='Fred Gary')c1.save()
(env)django-jsonapi-training$ ./manage.py testCreating test database for alias 'default'...System check identified no issues (0 silenced)..E..======================================================================ERROR: test_dup_fail (myapp.tests.test_models.CourseTestCase)----------------------------------------------------------------------Traceback (most recent call last): File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute return self.cursor.execute(sql, params) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 296, in execute return Database.Cursor.execute(self, query, params)sqlite3.IntegrityError: UNIQUE constraint failed: myapp_course.course_identifierThe above exception was the direct cause of the following exception:Traceback (most recent call last): File "/Users/alan/src/django-jsonapi-training/myapp/tests/test_models.py", line 114, in test_dup_fail last_mod_user_name='Fred Gary') File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/models/query.py", line 413, in create obj.save(force_insert=True, using=self.db) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/models/base.py", line 718, in save force_update=force_update, update_fields=update_fields) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/models/base.py", line 748, in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/models/base.py", line 831, in _save_table result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/models/base.py", line 869, in _do_insert using=using, raw=raw) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/models/query.py", line 1136, in _insert return query.get_compiler(using=using).execute_sql(return_id) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1289, in execute_sql cursor.execute(sql, params) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers return executor(sql, params, many, context) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute return self.cursor.execute(sql, params) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute return self.cursor.execute(sql, params) File "/Users/alan/src/django-jsonapi-training/env/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 296, in execute return Database.Cursor.execute(self, query, params)django.db.utils.IntegrityError: UNIQUE constraint failed: myapp_course.course_identifier----------------------------------------------------------------------Ran 4 tests in 0.025sFAILED (errors=1)Destroying test database for alias 'default'...
It's always good to write some negative tests to make sure expected errors actually occur. In
test_dup_fail() we see that it failed on a uniqueness constraint, which is an expected error. So let's update
the test code to check for the error:
GIT TAG: test-models-IntegrityError
1 2 3 4 5 6 7 8 9101112
deftest_dup_fail(self):withself.assertRaises(IntegrityError):c1=Course.objects.create(school_bulletin_prefix_code='123',suffix_two='11',subject_area_code='A101',course_number='123',course_identifier='ABC123',course_name='Corp. Finance',course_description='bla bla bla 123',last_mod_user_name='Fred Gary')c1.save()
123456789
(env)django-jsonapi-training$ ./manage.py testCreating test database for alias 'default'...System check identified no issues (0 silenced).....----------------------------------------------------------------------Ran 4 tests in 0.021sOKDestroying test database for alias 'default'...
PyCharm is an IDE for Python. There are both licensed and
community editions. I'm using the licensed edition
which has some Django-specific support, but even PyCharm CE can be used to develop and debug.
(If you are an
old-school CLI purist, I guess you could always use python -m pdb ./manage.py shell;-)
You can also set up the equivalent of ./manage.py runserver which I've found is very
handy for settting various environment variables. For example, I have both a sqlite3 and a sqlserver
"flavor" for running this project.
An especially useful enviroment variable to set from time-to-time (either in PyCharm or just
in the shell) is:
PYTHONWARNINGS=default since the default value for
PYTHONWARNINGS is ignore.
This helps find all those deprecation warnings that are otherwise not displayed:
Testing started at 3:20 PM .../Users/alan/src/django-training/env/bin/python /Applications/PyCharm.app/Contents/helpers/PyCharm/django_test_manage.py test /Users/alan/src/django-training/Users/ac45/.pyenv/versions/3.6.6/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /Users/alan/src/django-training/env/lib/python3.6/site-packages/aspy: missing __init__ _warnings.warn(msg.format(portions[0]), ImportWarning)Creating test database for alias 'default'...System check identified no issues (0 silenced)./Users/alan/src/django-training/myapp/tests/test_models.py:83: DeprecationWarning: Please use assertEqual instead. self.assertEquals(c.last_mod_date, date.today())/Users/alan/src/django-training/myapp/tests/test_models.py:87: DeprecationWarning: Please use assertEqual instead. self.assertEquals(2, cs.count())/Users/alan/src/django-training/myapp/tests/test_models.py:90: DeprecationWarning: Please use assertEqual instead. self.assertEquals(4, terms.count())/Users/alan/src/django-training/myapp/tests/test_models.py:94: DeprecationWarning: Please use assertEqual instead. self.assertEquals(0, ts.count())/Users/alan/src/django-training/myapp/tests/test_models.py:97: DeprecationWarning: Please use assertEqual instead. self.assertEquals(2, ts.count())/Users/alan/src/django-training/myapp/tests/test_models.py:100: DeprecationWarning: Please use assertEqual instead. self.assertEquals(2, ts.count())/Users/alan/src/django-training/myapp/tests/test_models.py:103: DeprecationWarning: Please use assertEqual instead. self.assertEquals(2, cs.count())/Users/alan/src/django-training/myapp/tests/test_models.py:78: DeprecationWarning: Please use assertEqual instead. self.assertEquals(c.course_identifier, 'ABC123')test_patch_primary_rel not yet implementedtest_patch_rel not yet implementedDestroying test database for alias 'default'...Process finished with exit code 0
This project has a much more complex set of test cases in
test_views.py.
A few things that happen here are:
Use fixtures for the test case data. Unlike the prior example where we explicitly did a bunch
of Course.objects.create() calls to create the test data, we instead load the data from a fixture.
Create a bunch of test users with various permissions (in preparation for doing some permission testing).
1 2 3 4 5 6 7 8 910111213141516171819202122
ClassDJATestCase(APITestCase):""" test cases using Django REST Framework drf-json-api (DJA) """fixtures=('testcases',)defsetUp(self):# define some usersself.superuser=User.objects.create_user('tester',is_superuser=True)# `somebody` can view course but not anything elseself.someuser=User.objects.create_user('somebody',is_superuser=False)self.someuser.user_permissions.add(Permission.objects.get(codename='view_course').id)# `nobody` has no permissionsself.noneuser=User.objects.create_user('nobody',is_superuser=False)# most tests just use the superuserself.client.force_authenticate(user=self.superuser)self.courses=Course.objects.all()self.courses_url=reverse('course-list')self.course_terms=CourseTerm.objects.all()self.course_terms_url=reverse('courseterm-list')# ...
At this point, two of the tests are skipped and labeled as such with @unittest.skip to remind me to write them.
Also, one test fails, exercising a bug in the current DJA 2.6.0 release. We'll get to how to deal with that
later.
Note that I've included running bandit and safety as part of the default tox test environment, so that
they always get run with tox. Some developers prefer to only run these via, e.g. tox -e bandit
(with a [testenv:bandit] section in the tox.ini). It really
depends on how long they take to run. For this tiny demonstration app, they make little difference and it's
best to always check this stuff.
Before we can use tox, a couple other things are needed:
- setup.py because tox requires it.
- README.md because our setup.py references it.
(env)django-jsonapi-training$ toxGLOB sdist-make: /Users/alan/src/django-jsonapi-training/setup.pypy36 create: /Users/alan/src/django-jsonapi-training/.tox/py36py36 bootstrap: venv-update>=2.1.3py36 installdeps: -rrequirements.txt,flake8,coverage,isortpy36 inst: /Users/alan/src/django-jsonapi-training/.tox/.tmp/package/1/myapp-0.1.0.zippy36 installed: You are using pip version 18.0, however version 18.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,certifi==2018.10.15,chardet==3.0.4,coverage==4.5.1,Django==2.1.2,django-admin==1.3.2,django-cors-middleware==1.3.1,django-debug-toolbar==1.10.1,django-excel-response2==2.0.8,django-filter==2.0.0,django-oauth-toolkit==1.2.0,django-six==1.0.4,djangorestframework==3.9.0,djangorestframework-jsonapi==2.6.0,filelock==3.0.9,flake8==3.6.0,idna==2.7,inflection==0.3.1,isort==4.3.4,mccabe==0.6.1,myapp==0.1.0,oauthlib==2.1.0,pluggy==0.8.0,py==1.7.0,pycodestyle==2.4.0,pyflakes==2.0.0,pytz==2018.6,PyYAML==3.13,requests==2.20.0,rest-condition==1.0.3,screen==1.0.1,six==1.11.0,sqlparse==0.2.4,toml==0.10.0,tox==3.5.2,tox-pip-extensions==1.4.1,urllib3==1.24,venv-update==3.1.1,virtualenv==16.0.0,xlwt==1.3.0py36 bootstrap: venv-update>=2.1.3py36 installdeps: -rrequirements.txt,flake8,coverage,isortpy36 installed: You are using pip version 18.0, however version 18.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,certifi==2018.10.15,chardet==3.0.4,coverage==4.5.1,Django==2.1.2,django-admin==1.3.2,django-cors-middleware==1.3.1,django-debug-toolbar==1.10.1,django-excel-response2==2.0.8,django-filter==2.0.0,django-oauth-toolkit==1.2.0,django-six==1.0.4,djangorestframework==3.9.0,djangorestframework-jsonapi==2.6.0,filelock==3.0.9,flake8==3.6.0,idna==2.7,inflection==0.3.1,isort==4.3.4,mccabe==0.6.1,myapp==0.1.0,oauthlib==2.1.0,pluggy==0.8.0,py==1.7.0,pycodestyle==2.4.0,pyflakes==2.0.0,pytz==2018.6,PyYAML==3.13,requests==2.20.0,rest-condition==1.0.3,screen==1.0.1,six==1.11.0,sqlparse==0.2.4,toml==0.10.0,tox==3.5.2,tox-pip-extensions==1.4.1,urllib3==1.24,venv-update==3.1.1,virtualenv==16.0.0,xlwt==1.3.0py36 run-test-pre: PYTHONHASHSEED='4264747819'py36 runtests: commands[0] | flake8 --exclude '**/migrations' myappmyapp/admin.py:1:1: F401 'django.contrib.admin' imported but unusedmyapp/tests/test_views.py:3:1: F401 'unittest.expectedFailure' imported but unusedERROR: InvocationError for command '/Users/alan/src/django-jsonapi-training/.tox/py36/bin/flake8 --exclude **/migrations myapp' (exited with code 1)_________________________________________________ summary __________________________________________________ERROR: py36: commands failed
Above we see a couple of flake8 errors that have to be fixed before the tests can be run.
1. myapp/admin.py (an django-admin startapp auto-generated file that we don't need) has an unused import.
2. myapp/tests/test_views.py also has an unused import.
3. Let's use that expectedFailure import and label the failing test so we can move on.
GIT TAG: tox-flake8-expectedFailure
1 2 3 4 5 6 7 8 9101112131415161718192021
diff --git a/myapp/admin.py b/myapp/admin.pyindex 8c38f3f..4185d36 100644--- a/myapp/admin.py+++ b/myapp/admin.py@@ -1,3 +1,3 @@-from django.contrib import admin+# from django.contrib import admin
# Register your models here.
diff --git a/myapp/tests/test_views.py b/myapp/tests/test_views.pyindex 698538b..c2dc37e 100644--- a/myapp/tests/test_views.py+++ b/myapp/tests/test_views.py@@ -384,6 +384,7 @@ class DJATestCase(APITestCase):
course_terms_data.append(course_term['data'])
self.assertEqual(course_terms_data, related['data'])
+ @expectedFailure
def test_related_course_terms_course(self):
"""
test toOne relationship and related links for course_terms.related.course
GLOB sdist-make: /Users/alan/src/django-jsonapi-training/setup.pypy36 inst-nodeps: /Users/alan/src/django-jsonapi-training/.tox/.tmp/package/1/myapp-0.1.0.zippy36 installed: You are using pip version 18.0, however version 18.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,certifi==2018.10.15,chardet==3.0.4,coverage==4.5.1,Django==2.1.2,django-admin==1.3.2,django-cors-middleware==1.3.1,django-debug-toolbar==1.10.1,django-excel-response2==2.0.8,django-filter==2.0.0,django-oauth-toolkit==1.2.0,django-six==1.0.4,djangorestframework==3.9.0,djangorestframework-jsonapi==2.6.0,filelock==3.0.9,flake8==3.6.0,idna==2.7,inflection==0.3.1,isort==4.3.4,mccabe==0.6.1,myapp==0.1.0,oauthlib==2.1.0,pluggy==0.8.0,py==1.7.0,pycodestyle==2.4.0,pyflakes==2.0.0,pytz==2018.6,PyYAML==3.13,requests==2.20.0,rest-condition==1.0.3,screen==1.0.1,six==1.11.0,sqlparse==0.2.4,toml==0.10.0,tox==3.5.2,tox-pip-extensions==1.4.1,urllib3==1.24,venv-update==3.1.1,virtualenv==16.0.0,xlwt==1.3.0py36 bootstrap: venv-update>=2.1.3py36 installdeps: -rrequirements.txt,flake8,coverage,isortpy36 installed: You are using pip version 18.0, however version 18.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,certifi==2018.10.15,chardet==3.0.4,coverage==4.5.1,Django==2.1.2,django-admin==1.3.2,django-cors-middleware==1.3.1,django-debug-toolbar==1.10.1,django-excel-response2==2.0.8,django-filter==2.0.0,django-oauth-toolkit==1.2.0,django-six==1.0.4,djangorestframework==3.9.0,djangorestframework-jsonapi==2.6.0,filelock==3.0.9,flake8==3.6.0,idna==2.7,inflection==0.3.1,isort==4.3.4,mccabe==0.6.1,myapp==0.1.0,oauthlib==2.1.0,pluggy==0.8.0,py==1.7.0,pycodestyle==2.4.0,pyflakes==2.0.0,pytz==2018.6,PyYAML==3.13,requests==2.20.0,rest-condition==1.0.3,screen==1.0.1,six==1.11.0,sqlparse==0.2.4,toml==0.10.0,tox==3.5.2,tox-pip-extensions==1.4.1,urllib3==1.24,venv-update==3.1.1,virtualenv==16.0.0,xlwt==1.3.0py36 run-test-pre: PYTHONHASHSEED='4222582931'py36 runtests: commands[0] | flake8 --exclude '**/migrations' myapppy36 runtests: commands[1] | isort -vb -df -c --recursive --skip migrations myapp/#######################################################################\ `sMMy` .yyyy- ` ##soos## ./o. ` ``..-..` ``...`.`` ` ```` ``-ssso``` .s:-y- .+osssssso/. ./ossss+:so+:` :+o-`/osso:+sssssssso/ .s::y- osss+.``.`` -ssss+-.`-ossso` ssssso/::..::+ssss:::. .s::y- /ssss+//:-.` `ssss+ `ssss+ sssso` :ssss` .s::y- `-/+oossssso/ `ssss/ sssso ssss/ :ssss` .y-/y- ````:ssss` ossso. :ssss: ssss/ :ssss. `/so:` `-//::/osss+ `+ssss+-/ossso: /sso- `osssso/. \/ `-/oooo++/- .:/++:/++/-` .. `://++/. isort your Python imports for you so you don't have to VERSION 4.3.4\########################################################################/else-type place_module for uuid returned STDLIBfrom-type place_module for django.db returned THIRDPARTYSUCCESS: /Users/alan/src/django-jsonapi-training/myapp/models.py Everything Looks Good!from-type place_module for datetime returned STDLIBfrom-type place_module for rest_framework_json_api.relations returned THIRDPARTYfrom-type place_module for rest_framework_json_api.serializers returned THIRDPARTYfrom-type place_module for myapp.models returned FIRSTPARTYSUCCESS: /Users/alan/src/django-jsonapi-training/myapp/serializers.py Everything Looks Good!WARNING: /Users/alan/src/django-jsonapi-training/myapp/__init__.py was skipped as it's listed in 'skip' setting or matches a glob in 'skip_glob' settingfrom-type place_module for django.apps returned THIRDPARTYSUCCESS: /Users/alan/src/django-jsonapi-training/myapp/apps.py Everything Looks Good!SUCCESS: /Users/alan/src/django-jsonapi-training/myapp/admin.py Everything Looks Good!from-type place_module for oauth2_provider.contrib.rest_framework returned THIRDPARTYfrom-type place_module for rest_condition returned THIRDPARTYfrom-type place_module for rest_framework.authentication returned THIRDPARTYfrom-type place_module for rest_framework.permissions returned THIRDPARTYfrom-type place_module for rest_framework_json_api.views returned THIRDPARTYfrom-type place_module for myapp.models returned FIRSTPARTYfrom-type place_module for myapp.serializers returned FIRSTPARTYSUCCESS: /Users/alan/src/django-jsonapi-training/myapp/views.py Everything Looks Good!WARNING: /Users/alan/src/django-jsonapi-training/myapp/tests/__init__.py was skipped as it's listed in 'skip' setting or matches a glob in 'skip_glob' settingelse-type place_module for json returned STDLIBelse-type place_module for math returned STDLIBfrom-type place_module for unittest returned STDLIBfrom-type place_module for django.contrib.auth.models returned THIRDPARTYfrom-type place_module for rest_framework.reverse returned THIRDPARTYfrom-type place_module for rest_framework.test returned THIRDPARTYfrom-type place_module for myapp.models returned FIRSTPARTYSUCCESS: /Users/alan/src/django-jsonapi-training/myapp/tests/test_views.py Everything Looks Good!from-type place_module for datetime returned STDLIBfrom-type place_module for django.db.utils returned THIRDPARTYfrom-type place_module for django.test returned THIRDPARTYfrom-type place_module for myapp.models returned FIRSTPARTYSUCCESS: /Users/alan/src/django-jsonapi-training/myapp/tests/test_models.py Everything Looks Good!WARNING: migrations was skipped as it's listed in 'skip' setting or matches a glob in 'skip_glob' settingSkipped 3 filespy36 runtests: commands[2] | python manage.py checkSystem check identified no issues (0 silenced).py36 runtests: commands[3] | bandit --recursive myapp[main] INFO profile include tests: None[main] INFO profile exclude tests: None[main] INFO cli include tests: None[main] INFO cli exclude tests: None[main] INFO running on Python 3.6.6Run started:2018-11-01 19:21:50.055332Test results: No issues identified.Code scanned: Total lines of code: 728 Total lines skipped (#nosec): 0Run metrics: Total issues (by severity): Undefined: 0.0 Low: 0.0 Medium: 0.0 High: 0.0 Total issues (by confidence): Undefined: 0.0 Low: 0.0 Medium: 0.0 High: 0.0Files skipped (0):py36 runtests: commands[4] | safety check --full-report╒══════════════════════════════════════════════════════════════════════════════╕│ ││ /$$$$$$ /$$ ││ /$$__ $$ | $$ ││ /$$$$$$$ /$$$$$$ | $$ \__//$$$$$$ /$$$$$$ /$$ /$$ ││ /$$_____/ |____ $$| $$$$ /$$__ $$|_ $$_/ | $$ | $$ ││ | $$$$$$ /$$$$$$$| $$_/ | $$$$$$$$ | $$ | $$ | $$ ││ \____ $$ /$$__ $$| $$ | $$_____/ | $$ /$$| $$ | $$ ││ /$$$$$$$/| $$$$$$$| $$ | $$$$$$$ | $$$$/| $$$$$$$ ││ |_______/ \_______/|__/ \_______/ \___/ \____ $$ ││ /$$ | $$ ││ | $$$$$$/ ││ by pyup.io \______/ ││ │╞══════════════════════════════════════════════════════════════════════════════╡│ REPORT ││ checked 54 packages, using default DB │╞══════════════════════════════════════════════════════════════════════════════╡│ No known security vulnerabilities found. │╘══════════════════════════════════════════════════════════════════════════════╛py36 runtests: commands[3] | coverage erasepy36 runtests: commands[4] | coverage run --source=/Users/alan/src/django-jsonapi-training/myapp manage.py testCreating test database for alias 'default'...System check identified no issues (0 silenced)...........ss...x...----------------------------------------------------------------------Ran 19 tests in 0.698sOK (skipped=2, expected failures=1)Destroying test database for alias 'default'...py36 runtests: commands[5] | coverage html_________________________________________________ summary __________________________________________________ py36: commands succeeded congratulations :)
One of the steps that ourt tox.ini performs is to generate a code coverage
report. After running tox, open htmlcov/index.html in your browser and
check out the coverage reports on which lines of code that
have been tested by the unit tests and which have been missed:
From this report we see that we could perhaps focus on improving
myapp/serializers.py. Let's take a look:
This highlights the fact that our tests are entirely missing any tests
involving the HyperlinkedModelSerializer.update() function, so perhaps
that's what the next test we write should do. In fact, this is not a surprise since
we have these two incomplete test functions in test_views.py:
1 2 3 4 5 6 7 8 910111213141516
fromunittestimportskip# ...@skip("test_patch_primary_rel not yet implemented")deftest_patch_primary_rel(self):""" TODO: I should be able to PATCH the primary data and updated relationships. """pass@skip("test_patch_rel not yet implemented")deftest_patch_rel(self):""" TODO: I should be able to PATCH the relationships. """pass
By the way, PyCharm's TODO view shows me any comments marked "TODO" (except those in .md files:-()
I also found that I have code that is never tested (myapp/apps.py).
This is another auto-generated file from django-admin startapp and is a candidate to
remove from the project.
Sometimes we'll run into a bug or want to use a new feature in one of the packages we rely on, that
is available but hasn't been released yet. It's actually pretty easy to do this via updates to
requirements.txt. Simply replace a package name and version dependency with a git reference.
A couple of real-world examples have to do with a race condition in DOT and a bug in DRF 2.6.0's
new RelatedMixin. Here's our modified requirements.txt that has a few changes from our simplistic
earlier version:
1. We are OK with any version of Django 2.1.x and matching versions of django-cors-middleware, etc.
2. We replace django-oauth-toolkit's latest published release (1.2.0) with a newer commit that has been
merged into the project but is not yet published.
3. We replace djangorestframework-jsonapi 2.6.0 with a commit of merged
PR that is not yet released.
4. All the customizations are commented so we know why they are there.
1 2 3 4 5 6 7 8 91011121314151617181920
Django>=2.1.0,<2.2django-filter==2.0.0django-cors-middlewaredjango-debug-toolbar# django-oauth-toolkit==1.2.0
# TODO: fix this when next release comes out
# fix duplicate null key error
git+https://github.com/n2ygk/django-oauth-toolkit.git@issue-663/duplicate_null_keydjango-pyodbc-azuredjangorestframework>=3.8,<3.9# TODO: fix this when next release comes out
# see https://github.com/django-json-api/django-rest-framework-json-api/pull/517
# djangorestframework-jsonapi==2.6.0
git+https://github.com/django-json-api/django-rest-framework-json-api.git@8c075d0requestsrest-condition# tox-pip-extensions breaks with tox 3.3+
tox>=3.3.0,<3.4tox-pip-extensions==1.4.0PyYAML
Let's clean up our world and try again:
1. Deactivate and remove the current virtualenv.
2. Use tox to set up our dev enviroment this time.
3. Run the tests and see we now have an unexpected success.
1 2 3 4 5 6 7 8 9101112131415161718192021222324
(env)django-jsonapi-training$ deactivatedjango-jsonapi-training$ rm -r env django-jsonapi-training$ tox -e devenvdevenv create: /Users/alan/src/django-jsonapi-training/envdevenv installdeps: -rrequirements.txtdevenv develop-inst: /Users/alan/src/django-jsonapi-trainingdevenv installed: ----------------------------------------,Error when trying to get requirement for VCS system Command "git config --get-regexp remote\..*\.url" failed with error code 1 in /Users/alan/src/django-jsonapi-training, falling back to uneditable format,Could not determine repository location of /Users/alan/src/django-jsonapi-training,certifi==2018.10.15,chardet==3.0.4,Django==2.1.2,django-admin==1.3.2,django-cors-middleware==1.3.1,django-debug-toolbar==1.10.1,django-excel-response2==2.0.8,django-filter==2.0.0,django-oauth-toolkit==1.2.0,django-pyodbc-azure==2.1.0.0,django-six==1.0.4,djangorestframework==3.8.2,djangorestframework-jsonapi==2.6.0,idna==2.7,inflection==0.3.1,## !! Could not determine repository location,myapp==0.1.0,oauthlib==2.1.0,pluggy==0.8.0,py==1.7.0,pyodbc==4.0.24,pytz==2018.7,PyYAML==3.13,requests==2.20.0,rest-condition==1.0.3,screen==1.0.1,six==1.11.0,sqlparse==0.2.4,toml==0.10.0,tox==3.3.0,tox-pip-extensions==1.4.0,urllib3==1.24,virtualenv==16.1.0,xlwt==1.3.0devenv runtests: PYTHONHASHSEED='2853687863'devenv runtests: commands[0] | /usr/bin/printf '\n\033[0;31m dont forget to source env/bin/activate\033[0m\n' dont forget to source env/bin/activate_________________________________________________ summary __________________________________________________ devenv: commands succeeded congratulations :)django-jsonapi-training$ source env/bin/activate(env)django-jsonapi-training$ ./manage.py testCreating test database for alias 'default'...System check identified no issues (0 silenced)...........ss...u...----------------------------------------------------------------------Ran 19 tests in 0.427sFAILED (skipped=2, unexpected successes=1)Destroying test database for alias 'default'...
Alternatively, run tox instead of ./manage.py test and see a little more verbose output
but ultimately the same success.
It's probably a better practice to pin all package versions (e.g. via pip freeze) to make sure your
production deployment is immutable.
1 2 3 4 5 6 7 8 9101112131415161718
Django==2.1.3django-filter==2.0.0django-cors-middleware==1.3.1django-debug-toolbar==1.9.1# django-oauth-toolkit==1.2.0
# master has my fix:
git+https://github.com/jazzband/django-oauth-toolkit.git@07f6430bdjango-pyodbc-azure==2.1.0.0djangorestframework==3.9.0# djangorestframework-jsonapi==2.6.0
# see https://github.com/django-json-api/django-rest-framework-json-api/pull/492
git+https://github.com/n2ygk/django-rest-framework-json-api.git@related_disable_pk_onlyrequests==2.19.1rest-condition==1.0.3# tox-pip-extensions breaks with tox 3.3+
tox>=3.3.0,<3.4tox-pip-extensions==1.4.0PyYAML==3.13
