Archives: August 6, 2012

SQLAlchemy Basic Setup

The project setup should have the following directories and files. I will go through the contents of the files shortly however for now lets just get the basic structure in place.

[project Folder]
|--app
|   |--__init__.py
|   |--models.py
|--config.py
|--main.py

Once the folder are created and the empty files are in place lets create the venv folder within the project and enter into the virtual environment. If you can’t remember how to do this there is a post about it here: python-venv-virtual-environments

Now lets install the python packages that we are going to be using:

(venv) $ pip install flask
(venv) $ pip install Flask-SQLAlchemy
(venv) $ pip install Flask-Migrate
(venv) $ pip install pyodbc

With everything being in place we can begin filling out the code required in each of the python files.

To keep things nicely organized, I’m going to create the configuration class in a separate Python module. Below you can see the new configuration class for this application, stored in a config.py module in the top-level directory.

/config.py

import os
basedir = os.path.abspath(os.path.dirname(file))

class Config:
    SECRET_KEY = 'you-will-never-guess'

    # This is the URI for SQLite
    SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or 'sqlite:///' +     os.path.join(basedir, 'app.db')

    # This is the URI for MS SQL
    SQLALCHEMY_DATABASE_URI =     'mssql+pyodbc://username:password@Server_Name:1433/DB_Name?     driver=ODBC+Driver+17+for+SQL+Server'

    # This is the URI for MS SQL
    SQLALCHEMY_DATABASE_URI = 'mysql://username:password@Server_Name/DB_Name'

The configuration settings are defined as class variables inside the Config class. Flask and some of its extensions use the value of the secret key as a cryptographic key, useful to generate signatures or tokens. The Flask-WTF extension uses it to protect web forms against Cross-Site Request Forgery or CSRF.

The next three lines starting with SQLALCHEMY_DATABASE_URI specifies the applications database connection. You will only need to use one of these connection they specify the connection strings for SQLite, MySQL and MS SQL.

app/init.py

from flask import Flask
from config import Config # Importing the Config class from the config.py file
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate

app = Flask(name)
app.config.from_object(Config)
db = SQLAlchemy(app)
migrate = Migrate(app, db)

from app import models

We first create the application object as an instance of class Flask imported from the flask package. We then import the config class from the config.py file, under which we import the SQLAlchemy and Migrate modules

The name variable passed to the Flask class is a Python predefined variable, which is set to the name of the module in which it is used. Flask uses the location of the module passed here as a starting point when it needs to load associated resources such as template files

We then pass in the Config class to the app created above.

We add a db object that represents the database. Then add migrate passing it the app and db objects, to represent the database migration engine.

Finally I importing a new module called models at the bottom. This module will define the structure of the database. The models module is imported at the bottom and not at the top of the script as it is always done. The bottom import is a workaround that avoids circular imports, a common problem with Flask applications.

/app/models.py

from datetime import datetime, timezone
from typing import Optional
import sqlalchemy as sa # General purpose database functions
import sqlalchemy.orm as so # Provides the support for using models
from app import db

class User(db.Model):
   id: so.Mapped[int] = so.mapped_column(primary_key=True)
   username: so.Mapped[str] = so.mapped_column(sa.String(64), index=True,   unique=True)
   email: so.Mapped[str] = so.mapped_column(sa.String(120), index=True, unique=True)
   password_hash: so.Mapped[Optional[str]] = so.mapped_column(sa.String(256))
   #posts: so.WriteOnlyMapped['Post'] = so.relationship(back_populates='author')

   def __repr__(self):
       return '<User {}>'.format(self.username)

We start by importing the datetime module which isn’t actually used in the class but your going to be righting stuff for a database so your going to be setting some timestamps which will probably be used as below so just import it:

timestamp: so.Mapped[datetime] = so.mapped_column(
        index=True, default=lambda: datetime.now(timezone.utc))

sqlalchemy and sqlalchemy.orm modules from the SQLAlchemy package, which provide most of the elements that are needed to work with a database. The sqlalchemy module includes general purpose database functions and classes such as types and query building helpers, while sqlalchemy.orm provides the support for using models. Given that these two module names are long and will need to be referenced often, the sa and so aliases are defined directly in the import statements. The db instance from Flask-SQLAlchemy and the Optional typing hint from Python are imported as well.

The User class created above will represent users stored in the database. The class inherits from db.Model, a base class for all models from Flask-SQLAlchemy. The User model defines several fields as class variables. These are the columns that will be created in the corresponding database table.

Fields are assigned a type using Python type hints, wrapped with SQLAlchemy’s so.Mapped generic type. A type declaration such as so.Mapped[int] or so.Mapped[str] define the type of the column, and also make values required, or non-nullable in database terms. To define a column that is allowed to be null, the Optional helper from Python is also added, as password_hash demonstrates.

In most cases defining a table column requires more than the column type. SQLAlchemy uses a so.mapped_column() function call assigned to each column to provide this additional configuration. In the case of id above, the column is configured as the primary key. For string columns many databases require a length to be given, so this is also included. I have included other optional arguments that allow us to indicate which fields are unique and indexed, which is important so that database is consistent and searches are efficient.

/main.py

import sqlalchemy as sa
import sqlalchemy.orm as so
from app import app, db
from app.models import User

@app.shell_context_processor
def make_shell_context():
   return {'sa': sa, 'so': so, 'db': db, 'user': User}

First we import the SQLAlchemy and SQLAlchemy.ORM modules as aliases in the same we we did in models. Flask application instance is called app and is a member of the app package. The from app import app statement imports the app variable that is a member of the app package as well as the db variable which is the database.

The app.shell_context_processor decorator registers the function as a shell context function. When the flask shell command runs, it will invoke this function and register the items returned by it in the shell session. The reason the function returns a dictionary and not a list is that for each item you have to also provide a name under which it will be referenced in the shell, which is given by the dictionary keys.

After you add the shell context processor function you can work with database entities without having to import them:

(venv) $ flask shell

>>>db
<SQLAlchemy mssql+pyodbc://Username:***@SQL_Server:1433/Test_Database?driver=ODBC+Driver+17+for+SQL+Server>
>>>
>>>User
<class 'app.models.User'>

Flask-Migrate

Flask-Migrate exposes its commands through the flask command. You have already seen flask run, which is a sub-command that is native to Flask. The flask db sub-command is added by Flask-Migrate to manage everything related to database migrations. So let’s create the migration repository for microblog by running flask db init:

(venv) $ flask db init
(venv) $ flask db migrate -m "users table"
(venv) $ flask db upgrade

CLI Database Interaction

To interact with the database from the python command line first run python

Import the application, database instance, models and SQLAlchemy entry point:

from app import app, db
from app.models import User
import sqlalchemy as sa

An application context must be created and pushed.

app.app_context().push()

Now we can create a new user:

u = User(username='Fin', email='fin@example.com')
db.session.add(u)
db.session.commit()

Changes to a database are done in the context of a database session, which can be accessed as db.session. Multiple changes can be accumulated in a session and once all the changes have been registered you can issue a single db.session.commit(), which writes all the changes atomically.

Let’s add another couple of user:

u = User(username='Eira', email='eira@example.com')
db.session.add(u)

u = User(username='Vix', email='vix@example.com')
db.session.add(u)

db.session.commit()

See how both the users where added to the database context and written to the database with the commit.

The database can answer a query that returns all the users.

query = sa.select(User)
users = db.session.scalars(query).all()
users

The query variable in this example is assigned a basic query that selects all the users. This is achieved by passing the model class to the SQLAlchemy sa.select() query helper function. You will find that most database queries start from a sa.select() call.

The database session, which above was used to define and commit changes, is also used to execute queries. The db.session.scalars() method executes the database query and returns a results iterator. Calling the all() method of the results object converts the results to a plain list.

In many situations it is most efficient to use the results iterator in a for-loop instead of converting it to a list:

users = db.session.scalars(query)
for u in users:
… print(u.id, u.username)
…

Python: venv (Virtual Environments)

Python virtual environments allows you load all of the dependencies a project may have into its own isolated environment. This means that if you have multiple projects with different versions of dependencies or wish to move your environment to a different location for example production. You will only need to move the dependencies required.

To setup a virtual environment perform the following steps:

  1. Ensure the Virtual Environments module is installed.
    pip install virtualenv
  2. To create a new virtual environment run the following command
    python -m venv C:\Scripts\project_Name\venv
  3. To activate the virtual environment navigate into the venv/Scripts folder created by the step above and execute activate
    activate
  4. To deactivate the environment simply type deactivate
    deactivate
  5. To export a list of requirements from your environment
    pip freeze --local > requirements.txt
  6. To install the exported requirements file
    pip install -r requirements.txt


Windows System Resource Manager

Open Server Manager and Select ‘Features’.  Scroll down to the ‘Windows System Resource Manager’ option and click to select.  If the ‘Add Features Wizard’ opens click the button to ‘Add Required Features’.

Click ‘Next’ and ‘Install’ to begin the installation.  Once the installation is complete click to close the install wizard.

To begin configuring Windows System Resource Manager navigate to Start, Administrative Tools, Windows System Resource Manager.

Ensure This computer is select and click ‘Connect’ in the Connect to computer dialog.

Expand Resource Allocation Policies and select ‘Equal_Per_Session’. On the right hand side click ‘Set as Managing Policy’.  Click ‘OK’ to the message “The calendar will be disabled.  Do you want to continue?”.


Connect Performance Monitor to a remote machine

Run Performance Monitor as a domain administrator. 

Click the + symbol to add a new performance counter.

Click the Browse button next to “Select counters from computer”

Enter the name of the computer you wish to monitor and click OK.

Press Enter on the keyboard.  There will be a slight delay while you are connected to the remote computer.  To confirm you have been connect the computer should now be in the drop down list.

Select each counter you wish to monitor and click add.  The counter(s) will appear in the Added Counters section on the right hand side with the name of the computer it is collecting information about next to it.

You will now begin to see the counter graphing the information.


Create a self-signed SSL Certificate – Linux

Step 1: Generate a Private Key

The openssl toolkit is used to generate an RSA Private Key and CSR (Certificate Signing Request). It can also be used to generate self-signed certificates which can be used for testing purposes or internal usage.

The first step is to create your RSA Private Key. This key is a 1024 bit RSA key which is encrypted using Triple-DES and stored in a PEM format so that it is readable as ASCII text.

Step 2: Generate a CSR (Certificate Signing Request)

Once the private key is generated a Certificate Signing Request can be generated. The CSR is then used in one of two ways. Ideally, the CSR will be sent to a Certificate Authority, such as Thawte or Verisign who will verify the identity of the requestor and issue a signed certificate. The second option is to self-sign the CSR, which is what I will be doing for this lab.

During the generation of the CSR, you will be prompted for several pieces of information. These are the X.509 attributes of the certificate. One of the prompts will be for “Common Name (e.g., YOUR name)”. It is important that this field be filled in with the fully qualified domain name of the server to be protected by SSL. If the website to be protected will be https://private.dlabs.co.uk, then enter private.dlabs.co.uk at this prompt. The command to generate the CSR is as follows:

openssl req -new -key server.key -out server.csr

Country Name (2 letter code) [GB]:GB
State or Province Name (full name) [Berkshire]:Berkshire
Locality Name (eg, city) [Newbury]:Newbury
Organization Name (eg, company) [My Company Ltd]:DLabs
Organizational Unit Name (eg, section) []:Information Technology
Common Name (eg, your name or your server's hostname) []:private.dlabs.co.uk
Email Address []:email.address@dlabs.co.uk
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Step 3: Remove Passphrase from Key

One unfortunate side-effect of the pass-phrased private key is that Apache will ask for the pass-phrase each time the web server is started. Obviously this is not necessarily convenient as someone will not always be around to type in the pass-phrase, such as after a reboot or crash. It is possible to remove the Triple-DES encryption from the key, thereby no longer needing to type in a pass-phrase. If the private key is no longer encrypted, it is critical that this file only be readable by the root user! If your system is ever compromised and a third party obtains your unencrypted private key, the corresponding certificate will need to be revoked. With that being said, use the following command to remove the pass-phrase from the key:

cp server.key server.key.org
openssl rsa -in server.key.org -out server.key

The newly created server.key file has no more passphrase in it.

-rw-r--r-- 1 root root 745 Jun 29 12:19 server.csr
-rw-r--r-- 1 root root 891 Jun 29 13:22 server.key
-rw-r--r-- 1 root root 963 Jun 29 13:22 server.key.org

Step 4: Generating a Self-Signed Certificate

At this point you will need to generate a self-signed certificate because you either don’t plan on having your certificate signed by a CA, or you wish to test your new SSL implementation while the CA is signing your certificate. This temporary certificate will generate an error in the client browser to the effect that the signing certificate authority is unknown and not trusted.

To generate a temporary certificate which is good for n days, issue the following command:

openssl x509 -req -days <number of days> -in server.csr -signkey server.key -out server.crt

Signature ok
subject=/C=UK/ST=Berkshire/L=Newbury/O=DLabs AG/OU=Information
Technology/CN=private.dlabs.co.uk/Email=email.address@dlabs.co.uk
Getting Private key

Step 5: Convert the PEM to PKCS
To convert the certificate and the key into one file that can be used in for example Firefox.

openssl pkcs12 -export -in input.crt -inkey input.key -certfile root.crt -out bundle.p12

Step 6: Installing the Private Key and Certificate

When Apache with mod_ssl is installed, it creates several directories in the Apache config directory. The location of this directory will differ depending on how Apache was compiled.

cp server.crt /usr/local/apache/conf/ssl.crt
cp server.key /usr/local/apache/conf/ssl.key

Step 7: Configuring SSL Enabled Virtual Hosts

SSLEngine on
SSLCertificateFile /usr/local/apache/conf/ssl.crt/server.crt
SSLCertificateKeyFile /usr/local/apache/conf/ssl.key/server.key
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
CustomLog logs/ssl_request_log \
   "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

Step 8: Restart Apache and Test

/etc/init.d/httpd stop
/etc/init.d/httpd stop

https://public.akadia.com

Windows Time Sources

Query the time source being used.

w32tm /query /status

Set the list of time sources

w32tm /config /manualpeerlist:<Time Source> /syncfromflags:MANUAL

update to use the new time source

w32tm /config /update

Query difference to time source.

w32tm /stripchart /computer:<Time Source> /samples:5 /dataonly
w32tm /stripchart /computer:chronos.csr.net /samples:5 /dataonly

Resync with the time source.

w32tm /resync

Generating files of any length

It is often useful when testing file transfers to have a file of a particular size. Rather than hunting around for a file of the required size why not simply create one of the desired size.

Generating files of any length in Windows

First open the command line interface by clicking Start > Run… and entering “cmd” (without the quotes) in the dialog form. By pressing Enter the command line interface will pop up and you can insert the following string to create a new file:

C:\>fsutil file createnew <filename> <filesize in bytes>

As you see you have to state the specific file size in bytes! For a conversion of megabytes or kilobytes to bytes see this or this conversion tool.

For example this string creates a new file named testfile.txt sized 1 Kb located in the root directory of partition C:

C:\>fsutil file createnew C:\testfile.txt 1024

Generating files of any length in Linux

File generation with Linux is as easy as with Windows. The `dd` tool to (amongst others) create new files comes with virtually every distribution. Here is the example command, intended to be run from within a shell.

dd if=/dev/zero of=<filename> bs=<initial blocksize in bytes> count=<iterations of the blocksize>

The easiest way to create a file of specific length using `dd`is by utilizing suffixes like K (for Kilobytes) or M (for Megabytes) like this:

dd if=/dev/zero of=testfile.txt bs=1K count=1

The command above creates a file of 1KB size in the current working directory.

The man page of `dd`lists the suffixes you may utilize:

BLOCKS and BYTES may be followed by the following multiplicative suffixes: xM M, c 1, w 2, b 512, kB 1000, K 1024, MB 1000*1000, M 1024*1024, GB 1000*1000*1000, G 1024*1024*1024, and so on for T, P, E, Z, Y.

As `dd` is available for all Linux/Unix distributions this applies to Unix Systems (e.g. Solaris) as well.


Cookie Consent Banner by Real Cookie Banner