sudo apt update
sudo apt install figlet
#!/bin/bash
# Define colors for printouts
GREEN="\033[1;32m"
BLUE="\033[1;34m"
YELLOW="\033[1;33m"
RESET="\033[0m"
CYAN="\033[1;36m"
MAGENTA="\033[1;35m"
# Impressive entry dialog
clear
echo -e "${BLUE}"
figlet -c 'PARKLIZE'
echo -e "${RESET}"
sleep 3 # Pause for a few seconds to allow the user to read the banner
In order to check the list of Java versions available on our Linux machine, we can use the following command.
$ sudo update-alternatives --config java
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-17-openjdk-amd64/bin/java 1711 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
* 2 /usr/lib/jvm/java-17-openjdk-amd64/bin/java 1711 manual mode
3 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
$ java --version
openjdk 17.0.10 2024-01-16
OpenJDK Runtime Environment (build 17.0.10+7-Ubuntu-122.04.1)
OpenJDK 64-Bit Server VM (build 17.0.10+7-Ubuntu-122.04.1, mixed mode, sharing)
Apache Jena is a free and open source Java framework for building Semantic Web and Linked Data applications. The framework is composed of different APIs interacting together to process RDF data, as we can see from the figure below.
The focus of this post is about setting up Fuseki using Docker. In short, Fuseki a SPARQL server which can present RDF data and answer SPARQL queries over HTTP.
The content of this post is as follows:
If you are familiar with Docker, it is very convinient to set up a Fuseki server using a Fuseki Docker image from the Docker Hub, which is a cloud-based repository service provided by Docker for finding, storing, and sharing container images.
I have Docker Desktop in my laptop which provides an integrated development environment (IDE) for building, shipping, and running containerized applications using Docker.
After pulling the Fuseki Docker image, we can follow the instruction to run a Fuseki server already:
docker run --rm -it -p 3030:3030 --name fuseki -e ADMIN_PASSWORD=[PASSWORD] -e ENABLE_DATA_WRITE=[true|false] -e ENABLE_UPDATE=[true|false] -e QUERY_TIMEOUT=[number in milliseconds] --mount type=bind,source="$(pwd)"/fuseki-data,target=/fuseki-base/databases secoresearch/fuseki
mkdir fuseki-data
mkdir fuseki-configuration
cp -p assembler.ttl fuseki-configuration/
# edit fuseki-configuration/assembler.ttl to enable the endpoints you wish
docker run --rm -it -p 3030:3030 --name fuseki -e ADMIN_PASSWORD=[PASSWORD] -e QUERY_TIMEOUT=[number in milliseconds] --mount type=bind,source="$(pwd)"/fuseki-data,target=/fuseki-base/databases --mount type=bind,source="$(pwd)"/fuseki-configuration,target=/fuseki-base/configuration secoresearch/fuseki
<#service> rdf:type fuseki:Service ;
fuseki:name "ds" ; # http://host:port/ds
fuseki:dataset <#tdb> ;
fuseki:endpoint [
# SPARQL query service
fuseki:operation fuseki:query ;
fuseki:name "sparql"
] ;
... ...
<#tdb> rdf:type ja:RDFDataset ;
rdfs:label "EnergyConsumption" ;
ja:defaultGraph
[ rdfs:label "DAYTON.ttl" ;
a ja:MemoryModel ;
ja:content [ja:externalContent <ttl file location>] ;
] ;
.
If you are planning to interact with the Fuseki server set up in your program such as using Python, you might need to test SPARQL queries via HTTP out first. One way is directly using the browser and type your endpoint with your query parameter:
http://localhost:3030/ds/sparql?query=SELECT%20*%20WHERE%20{?s%20?p%20?o}%20limit%203
If you are using curl for testing a SPARQL query, you can submit a URL-encoded SPARQL query
curl "http://localhost:3030/ds/sparql?query=SELECT%20*%20WHERE%20\{?s%20?p%20?o\}%20limit%203"
It seems escapes for starting and ending brackets are needed. For more details regarding using curl for SPARQL queries, one can refern to cURLing SPARQL, which contains much more details regarding the topic.
openapi: 4.0.2
info:
title: Sample OpenAPI Specification
description: 'An OpenAPI specification example for Building API services: A Beginners Guide document.'
version: 0.0.1
servers:
- url: http://localhost:9000/
description: Example API Service
components:
schemas:
'User':
type: object
required:
- display_name
- email
properties:
name:
type: string
readOnly: true
display_name:
type: string
maxLength: 20
minLength: 1
email:
type: string
format: email
'ErrorMessage':
type: object
required:
- error_code
- error_message
properties:
error_code:
type: string
error_message:
type: string
paths:
/users/{user_id}:
parameters:
- name: user_id
in: path
description: ID of a user
required: true
schema:
type: string
get:
description: Gets a user
operationId: get_user
responses:
'200':
description: User found
content:
'application/json':
schema:
$ref: '#/components/schemas/User'
'default':
description: Unexpected error
content:
'application/json':
schema:
$ref: '#/components/schemas/ErrorMessage'
$openapi-generator-cli version
7.2.0
openapi-generator-cli generate -i openapi.yaml -o generated -g python-flask
Dockerfile
git_push.sh
/openapi_server
/controllers
/models
/test
/openapi
encoder.py
__init__.py
__main__.py
__pycache__
typing_utils.py
util.py
README.md
requirements.txt
setup.py
test-requirements.txt
tox.ini
pip3 install -r requirements.txt
1 connexion[swagger-ui] >= 2.6.0; python_version>="3.6"
2 # 2.3 is the last version that supports python 3.4-3.5
3 connexion[swagger-ui] <= 2.3.0; python_version=="3.5" or python_version=="3.4"
4 # connexion requires werkzeug but connexion < 2.4.0 does not install werkzeug
5 # we must peg werkzeug versions below to fix connexion
6 # https://github.com/zalando/connexion/pull/1044
7 werkzeug == 0.16.1; python_version=="3.5" or python_version=="3.4"
8 swagger-ui-bundle >= 0.0.2
9 python_dateutil >= 2.6.0
10 setuptools >= 21.0.0
11 Flask == 2.1.1
1 connexion[swagger-ui,flask] >= 2.6.0; python_version>="3.6"
2 # 2.3 is the last version that supports python 3.4-3.5
3 connexion[swagger-ui] <= 2.3.0; python_version=="3.5" or python_version=="3.4"
4 # connexion requires werkzeug but connexion < 2.4.0 does not install werkzeug
5 # we must peg werkzeug versions below to fix connexion
6 # https://github.com/zalando/connexion/pull/1044
7 werkzeug == 0.16.1; python_version=="3.5" or python_version=="3.4"
8 swagger-ui-bundle >= 0.0.2
9 python_dateutil >= 2.6.0
10 setuptools >= 21.0.0
from json import JSONEncoder
from openapi_server.models.base_model import Model
class JSONEncoder(JSONEncoder):
include_nulls = False
def default(self, o):
if isinstance(o, Model):
dikt = {}
for attr in o.openapi_types:
value = getattr(o, attr)
if value is None and not self.include_nulls:
continue
attr = o.attribute_map[attr]
dikt[attr] = value
return dikt
return JSONEncoder.default(self, o)
python3 -m openapi_server
def get_user(user_id): # noqa: E501
"""get_user
Gets a user # noqa: E501
:param user_id: ID of a user
:type user_id: str
:rtype: Union[User, Tuple[User, int], Tuple[User, int, Dict[str, str]]
"""
return 'do some magic!'
I was wondering the difference between FILTER NOT EXISTS and MINUS, and found out a great answer here
The difference between FILTER NOT EXISTS and MINUS is related to the two styles of negation used by SPARQL. According to the specification:
The SPARQL query language incorporates two styles of negation, one based on filtering results depending on whether a graph pattern does or does not match in the context of the query solution being filtered, and one based on removing solutions related to another pattern.
Still according to the specification:
NOT EXISTS and MINUS represent two ways of thinking about negation, one based on testing whether a pattern exists in the data, given the bindings already determined by the query pattern, and one based on removing matches based on the evaluation of two patterns. In some cases they can produce different answers.
The two requests of your question are cited in the specification and the results are explained in the following way:
SELECT * {
?s ?p ?o .
FILTER NOT EXISTS { ?x ?y ?z } .
}
This request evaluates to a result set with no solutions because { ?x ?y ?z } matches given any ?s ?p ?o, so NOT EXISTS { ?x ?y ?z } eliminates any solutions.
SELECT * {
?s ?p ?o .
MINUS { ?x ?y ?z } .
}
In the request with MINUS, there is no shared variable between the first part (?s ?p ?o) and the second (?x ?y ?z) so no bindings are eliminated.
$ vim ~/anaconda3/envs/pymc_env/lib/python3.10/site-packages/theano/link/c/cmodule.py:2621
2621 #blas_info = numpy.distutils.__config__.blas_opt_info
2622 blas_info = numpy.__config__.blas_opt_info
import tensorflow as tf
import tensorflow_probability as tfp
import seaborn as sns
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Flatten, Dense, Reshape
print("tensorflow", tf.__version__)
print("tensorflow probability", tfp.__version__)
print("matplotlib", matplotlib.__version__)
print("numpy", np.__version__)
print("seaborn", sns.__version__)
tfd = tfp.distributions
tfpl = tfp.layers
tensorflow 2.8.0 tensorflow probability 0.14.0 matplotlib 3.8.0 numpy 1.26.0 seaborn 0.13.0
# Fashion MNIST dataset
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
# If using Bernoulli, we can scale by simply dividing the max value
#X_train = X_train.astype('float32')/256.
#X_test = X_test.astype('float32')/256.
# If we use Beta distribution defined within interval (0,1),
# we need to scale images to the range (avoiding 0)
X_train = X_train.astype('float32')/256. + 0.5/256
X_test = X_test.astype('float32')/256. + 0.5/256
print(X_train.shape)
class_names = np.array([
'T-shirt/top',
'Trouser/pants',
'Pullover shirt',
'Dress',
'Coat',
'Sandal',
'Shirt',
'Sneaker',
'Bag',
'Ankle boot'
])
# Show some examples of the data
n_examples = 1000
example_images = X_test[0:n_examples]
example_labels = y_test[0:n_examples]
fig, axes = plt.subplots(1, 5, figsize=(15, 4))
for i in range(len(axes)):
axes[i].imshow(example_images[i], cmap='binary')
axes[i].set_title(class_names[example_labels[i]])
axes[i].axis('off')
encoded_dim = 2
# Identity covariance matrix by default
prior = tfd.MultivariateNormalDiag(
loc=tf.zeros(encoded_dim)
)
KLDivergenceAddLoss is a bypass layer, but will automatically add KL divergence loss between the prior and posterior to the main loss - the reconstruction loss.
# Encoder version 1
# Feel free to skip other versions and come back later
encoder = Sequential([
Flatten(input_shape=(28, 28)),
Dense(256, activation='relu'),
Dense(64, activation='relu'),
Dense(tfpl.MultivariateNormalTriL.params_size(encoded_dim)),
tfpl.MultivariateNormalTriL(encoded_dim),
tfpl.KLDivergenceAddLoss(prior)
])
print(encoder.losses, end='\n\n')
print(encoder(example_images), end='\n\n')
print(encoder.losses)
[tf.Tensor 'kl_divergence_add_loss_1/kldivergence_loss/batch_total_kl_divergence:0' shape=() dtype=float32]
tfp.distributions._TensorCoercible("sequential_2_multivariate_normal_tri_l_1_tensor_coercible", batch_shape=[1000], event_shape=[2], dtype=float32)
[tf.Tensor: shape=(), dtype=float32, numpy=0.89796853]
KLDivergenceAddLoss.
# Encoder version 2: with some arguments
encoder = Sequential([
Flatten(input_shape=(28, 28)),
Dense(256, activation='relu'),
Dense(64, activation='relu'),
Dense(tfpl.MultivariateNormalTriL.params_size(encoded_dim)),
tfpl.MultivariateNormalTriL(encoded_dim),
tfpl.KLDivergenceAddLoss(
prior,
use_exact_kl=False,
weight=1.5,
test_points_fn=lambda d: d.sample(10),
test_points_reduce_axis=0
)
])
weight: what multiple of KL divergence to be added to the loss, useful for implementing 𝛽-VAE, where 𝛽 indicates the weight of KL divergence.
test_points_fn: Receives batch of distributions, returns tensor of samples of shape (n_sample, batch_size, dim_z). These samples are converted to scalar value. $z_{ij}$ is the $i$-th sample for the observation $x_j$ (is at (i,j,:) in the tensor of samples) is mapped to $\log q(z_{ij})|x_j-\log p(z_{ij})$. This implies the tensor of samples returned by test_points_fn is converted into a tensor of values with a shape (n_samples, batch)
test_points_reduce_axis: to compute the loss added to the model, this arg indicates axis to average over (reduce_mean)
KLDivergenceAddLoss. Instead, we can declare the KLDivergenceRegularizer as below and specify the activity_regularizer parameter in the MultivariateNormalTriL directly.
# Encoder version 3: Using KLDivergenceRegularizer
divergence_regularizer = tfpl.KLDivergenceRegularizer(
prior,
use_exact_kl=False,
test_points_fn=lambda d: d.sample(10),
test_points_reduce_axis=0
)
encoder = Sequential([
Flatten(input_shape=(28, 28)),
Dense(256, activation='relu'),
Dense(64, activation='relu'),
Dense(tfpl.MultivariateNormalTriL.params_size(encoded_dim)),
tfpl.MultivariateNormalTriL(
encoded_dim,
activity_regularizer=divergence_regularizer
),
])
pretrain_example_encodings = encoder(example_images).mean().numpy()
# Plot encoded examples before training
f, ax = plt.subplots(1, 1, figsize=(7, 7))
sns.scatterplot(x=pretrain_example_encodings[:, 0],
y=pretrain_example_encodings[:, 1],
hue=class_names[example_labels], ax=ax,
palette=sns.color_palette("colorblind", 10));
ax.set_xlabel('Encoding dimension 1'); ax.set_ylabel('Encoding dimension 2')
ax.set_title('Encodings of example images before training')
IndependentBernoulli layer which we can directly use for the first version. For the second one, as there is no such independent Beta layer implemented, we bake one from scratch using the DistributionLambda layer.
# Decoder version 1: Using IndependentBernoulli
decoder = Sequential([
Dense(64, activation='relu', input_shape=(encoded_dim,)),
Dense(256, activation='relu'),
Dense(28*28),
tfpl.IndependentBernoulli((28, 28))
])
# Decoder version 2: Using Independent Beta Distribution
# Since there is no IndependentBeta layer, bake one from scratch
decoder = Sequential([
Dense(64, activation='relu', input_shape=(encoded_dim,)),
Dense(256, activation='relu'),
Dense(28*28*2, activation='exponential'), # non-nengative for Beta distribution params
Reshape((28, 28, 2)),
tfpl.DistributionLambda(
lambda t: tfd.Independent(
tfd.Beta(
concentration1=t[..., 0],
concentration0=t[..., 1]
)
)
)
])
KLDivergenceAddLoss layer already automatically add the KL/regularization loss to the main loss. Here we only need to specify the log loss.
vae = Model(
inputs=encoder.inputs,
outputs=decoder(encoder.outputs)
)
def log_loss(x_true, p_x_given_z):
return -tf.reduce_sum(p_x_given_z.log_prob(x_true))
vae.compile(loss=log_loss,)
vae.fit(
x=X_train,
y=X_train,
validation_data=(X_test, X_test),
epochs=10,
batch_size=32
)
WARNING:tensorflow:@custom_gradient grad_fn has 'variables' in signature, but no ResourceVariables were used on the forward pass.
WARNING:tensorflow:@custom_gradient grad_fn has 'variables' in signature, but no ResourceVariables were used on the forward pass.
WARNING:tensorflow:@custom_gradient grad_fn has 'variables' in signature, but no ResourceVariables were used on the forward pass.
WARNING:tensorflow:@custom_gradient grad_fn has 'variables' in signature, but no ResourceVariables were used on the forward pass.
WARNING:tensorflow:@custom_gradient grad_fn has 'variables' in signature, but no ResourceVariables were used on the forward pass.
1874/1875 [============================>.] - ETA: 0s - loss: -55572.9219WARNING:tensorflow:@custom_gradient grad_fn has 'variables' in signature, but no ResourceVariables were used on the forward pass.
WARNING:tensorflow:@custom_gradient grad_fn has 'variables' in signature, but no ResourceVariables were used on the forward pass.
WARNING:tensorflow:@custom_gradient grad_fn has 'variables' in signature, but no ResourceVariables were used on the forward pass.
1875/1875 [==============================] - 34s 16ms/step - loss: -55578.1367 - val_loss: -62465.1016
Epoch 2/10
1875/1875 [==============================] - 28s 15ms/step - loss: -64829.3750 - val_loss: -65042.4648
Epoch 3/10
1875/1875 [==============================] - 29s 15ms/step - loss: -67461.9844 - val_loss: -70393.7500
Epoch 4/10
1875/1875 [==============================] - 29s 15ms/step - loss: -69034.2734 - val_loss: -66066.8672
Epoch 5/10
1875/1875 [==============================] - 29s 15ms/step - loss: -70065.8438 - val_loss: -68278.2109
Epoch 6/10
1875/1875 [==============================] - 29s 15ms/step - loss: -70965.3438 - val_loss: -69826.4219
Epoch 7/10
1875/1875 [==============================] - 29s 15ms/step - loss: -71667.5000 - val_loss: -72257.3984
Epoch 8/10
1875/1875 [==============================] - 29s 15ms/step - loss: -72232.3594 - val_loss: -68337.1094
Epoch 9/10
1875/1875 [==============================] - 29s 16ms/step - loss: -72632.1641 - val_loss: -71673.4922
Epoch 10/10
1875/1875 [==============================] - 30s 16ms/step - loss: -72972.3281 - val_loss: -72791.6250
# Generate an example reconstruction
example_reconstruction = vae(example_images).mean().numpy().squeeze()
# Plot the example reconstructions
fig, axs = plt.subplots(2, 6, figsize=(16, 5))
for j in range(6):
axs[0, j].imshow(example_images[j, :, :].squeeze(), cmap='binary')
axs[1, j].imshow(example_reconstruction[j, :, :], cmap='binary')
axs[0, j].axis('off')
axs[1, j].axis('off')
# Compute example encodings after training
posttrain_example_encodings = encoder(example_images).mean().numpy()
# Compare the example encodings before and after training
f, axs = plt.subplots(nrows=1, ncols=2, figsize=(15, 7))
sns.scatterplot(
x=pretrain_example_encodings[:, 0],
y=pretrain_example_encodings[:, 1],
hue=class_names[example_labels], ax=axs[0],
palette=sns.color_palette("colorblind", 10)
)
sns.scatterplot(
x=posttrain_example_encodings[:, 0],
y=posttrain_example_encodings[:, 1],
hue=class_names[example_labels],
ax=axs[1],
palette=sns.color_palette("colorblind", 10)
)
axs[0].set_title('Encodings of example images before training');
axs[1].set_title('Encodings of example images after training');
for ax in axs:
ax.set_xlabel('Encoding dimension 1')
ax.set_ylabel('Encoding dimension 2')
ax.legend(loc='upper right')
import requests
url = 'http://example.com/'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
result = requests.get(url, headers=headers)
# Check the status code
print(result.status_code)
"WARNING:tensorflow:Early stopping conditioned on metric `val_loss` which is not available. Available metrics are: loss,mae,mse,mape"
This is normally due to the validation dataset is empty, so there is no associated val_loss for the empty validation data.
Windowed dataset is required in many forecasting methods, especially for machine learning-based approaches. As an example, given a time series, e.g., [1,3,5,7,9], and a window size of 2, the corresponding windowed dataset could be as follows, which can be used for training a machine learning model where each element in X represents windowed data while each element in y represents a corresponding label/target/next value to be predicted.
X = [[1,3], [3,5], [5,7]] y = [[5], [7], [9]]
import tensorflow as tf
import numpy as np
print(tf.__version__)
print(np.__version__)
2.4.1 1.19.2
For simplicity, here we create a sequence of numbers as our example time series data.
ts = np.arange(1, 100, 2)
The window_dataset() below provides a functionality to produce the windowed dataset for a given time series (ts) and the window_size. We delve into some details of the code in the following.
def window_dataset(ts, window_size=2):
""" Process the time series into windowed dataset
:parameter ts: time series data
Return data, targets where data and targets are both list
each element in data is history in a window,
each element in targets is the next value
"""
data = list()
targets = list()
dataset = tf.data.Dataset.from_tensor_slices(ts)
dataset = dataset.window(
window_size+1,
shift=1,
drop_remainder=True
)
dataset = dataset.flat_map(lambda w: w.batch(window_size+1))
dataset = dataset.map(lambda w: (w[:-1], w[-1:]))
for (x, y) in dataset.as_numpy_iterator():
data.append(x)
targets.append(y)
return data, targets
data, targets = window_dataset(ts)
for x,y in zip(data, targets):
print(x,y)
[1 3] [5] [3 5] [7] [5 7] [9] [7 9] [11] [ 9 11] [13] [11 13] [15] [13 15] [17] [15 17] [19] [17 19] [21] [19 21] [23] [21 23] [25] [23 25] [27] [25 27] [29] [27 29] [31] [29 31] [33] [31 33] [35] [33 35] [37] [35 37] [39] [37 39] [41] [39 41] [43] [41 43] [45] [43 45] [47] [45 47] [49] [47 49] [51] [49 51] [53] [51 53] [55] [53 55] [57] [55 57] [59] [57 59] [61] [59 61] [63] [61 63] [65] [63 65] [67] [65 67] [69] [67 69] [71] [69 71] [73] [71 73] [75] [73 75] [77] [75 77] [79] [77 79] [81] [79 81] [83] [81 83] [85] [83 85] [87] [85 87] [89] [87 89] [91] [89 91] [93] [91 93] [95] [93 95] [97] [95 97] [99]
from sklearn.linear_model import LinearRegression
model = LinearRegression(fit_intercept=True)
model.fit(data, targets)
model.predict([[97,99]])
array([[101.]])
Now, we move on to some details of the window_dataset() method. First step is creating a Tensorflow dataset. The from_tensor_slices(ts) creates a Dataset whose elements are slices of the given tensors.
dataset = tf.data.Dataset.from_tensor_slices(ts)
for d in dataset:
print(d)
break
tf.Tensor(1, shape=(), dtype=int64)
The Tensorflow dataset provides a method - window() - which a dataset of "windows". According to the documentation, each "window" is a dataset that contains a subset of elements of the input dataset. These are finite datasets of size size (or possibly fewer if there are not enough input elements to fill the window and drop_remainder evaluates to False). As the subset is still a dataset, we use list() and as_numpy_iterator() which returns an iterator which converts all elements of the dataset to numpy.
window_size = 2
dataset = dataset.window(
window_size+1,
shift=1,
drop_remainder=True
)
for d in dataset:
# Each d will be sub-dataset of the dataset
print(list(d.as_numpy_iterator()))
break
[1, 3, 5]
Here, we grab all data in each sub-dataset and flattens the result.
# Maps .batch across each sub-dataset
dataset = dataset.flat_map(lambda w: w.batch(window_size+1))
for d in dataset:
print(d)
break
tf.Tensor([1 3 5], shape=(3,), dtype=int64)
Finally, we split the data part and target one for each element in the dataset, and each pair will be a training example for training a forecasting model.
dataset = dataset.map(lambda w: (w[:-1], w[-1:]))
for (x, y) in dataset.as_numpy_iterator():
print(x, y)
break
[1 3] [5]That's it for preparing windowed dataset for time series forecasting. Although we used Tensorflow to implement the preprocessing step, you can also try to use other ways to implement the same functionality as long as you can derive the same output.
import tensorflow
import matplotlib
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Flatten, Reshape
print(tensorflow.__version__)
print(matplotlib.__version__)
print(np.__version__)
print(sns.__version__)
print(matplotlib.__version__)
# Load Fashion MNIST
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
x_train = x_train.astype('float32')/255.
x_test = x_test.astype('float32')/255.
class_names = np.array([
'T-shirt/top',
'Trouser/pants',
'Pullover shirt',
'Dress',
'Coat',
'Sandal',
'Shirt',
'Sneaker',
'Bag',
'Ankle boot'
])
print(x_train.shape)
We can have a look on some of those images.
# Display a few examples
n_examples = 1000
example_images = x_test[0:n_examples]
example_labels = y_test[0:n_examples]
f, axs = plt.subplots(1, 5, figsize=(15, 4))
for j in range(len(axs)):
axs[j].imshow(example_images[j], cmap='binary')
axs[j].axis('off')
Now we move on to the implementation of the encoder part of Autoencoder. The encoder simply flattens the input image and goes through two Dense layers followed by another Dense layer with desired encoding dimensionality, which is 2 here.
We can check the compressed or encoded images using this encoder. Note as the encoder has not been trained yet, we should see those encoded images from different class are not distinguishable in the encoding space.
# Define the encoder
encoded_dim = 2
encoder = Sequential([
Flatten(input_shape=(28, 28)),
Dense(256, activation='sigmoid'),
Dense(64, activation='sigmoid'),
Dense(encoded_dim)
])
# Encode examples before training
pretrain_example_encodings = encoder(example_images).numpy()
# Plot encoded examples before training
f, ax = plt.subplots(1, 1, figsize=(7, 7))
sns.scatterplot(pretrain_example_encodings[:, 0],
pretrain_example_encodings[:, 1],
hue=class_names[example_labels], ax=ax,
palette=sns.color_palette("colorblind", 10));
ax.set_xlabel('Encoding dimension 1'); ax.set_ylabel('Encoding dimension 2')
ax.set_title('Encodings of example images before training');
# Define the decoder
decoder = Sequential([
Dense(64, activation='sigmoid', input_shape=(encoded_dim,)),
Dense(256, activation='sigmoid'),
Dense(28*28, activation='sigmoid'),
Reshape((28, 28))
])
# Compile and fit the model
autoencoder = Model(
inputs=encoder.input,
outputs=decoder(encoder.output)
)
# Specify loss - input and output is in [0., 1.], so we can use a binary cross-entropy loss
autoencoder.compile(loss='binary_crossentropy')
# Fit model - highlight that labels and input are the same
autoencoder.fit(
x=x_train,
y=x_train,
epochs=10,
batch_size=32
)
# Compute example encodings after training
posttrain_example_encodings = encoder(example_images).numpy()
# Compare the example encodings before and after training
f, axs = plt.subplots(nrows=1, ncols=2, figsize=(15, 7))
sns.scatterplot(pretrain_example_encodings[:, 0],
pretrain_example_encodings[:, 1],
hue=class_names[example_labels], ax=axs[0],
palette=sns.color_palette("colorblind", 10));
sns.scatterplot(posttrain_example_encodings[:, 0],
posttrain_example_encodings[:, 1],
hue=class_names[example_labels], ax=axs[1],
palette=sns.color_palette("colorblind", 10));
axs[0].set_title('Encodings of example images before training');
axs[1].set_title('Encodings of example images after training');
for ax in axs:
ax.set_xlabel('Encoding dimension 1')
ax.set_ylabel('Encoding dimension 2')
ax.legend(loc='upper right')
# Compute the autoencoder's reconstructions
reconstructed_example_images = autoencoder(example_images)
# Evaluate the autoencoder's reconstructions
f, axs = plt.subplots(2, 5, figsize=(15, 4))
for j in range(5):
axs[0, j].imshow(example_images[j], cmap='binary')
axs[1, j].imshow(reconstructed_example_images[j].numpy().squeeze(), cmap='binary')
axs[0, j].axis('off')
axs[1, j].axis('off')
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
####################################################
As suggested by the error message itself, we just need to downgrade the protobuf package to 3.20.x or lower.
2023-10-19 10:01:41.895250: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-10-19 10:01:41.895278: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
File "t01.py", line 7, in <module>
import tensorflow.compat.v2 as tf
File "/home/parklize/Documents/code/tfp/venv/lib/python3.8/site-packages/tensorflow/__init__.py", line 37, in <module>
from tensorflow.python.tools import module_util as _module_util
File "/home/parklize/Documents/code/tfp/venv/lib/python3.8/site-packages/tensorflow/python/__init__.py", line 37, in <module>
from tensorflow.python.eager import context
File "/home/parklize/Documents/code/tfp/venv/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 29, in <module>
from tensorflow.core.framework import function_pb2
File "/home/parklize/Documents/code/tfp/venv/lib/python3.8/site-packages/tensorflow/core/framework/function_pb2.py", line 16, in <module>
from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
File "/home/parklize/Documents/code/tfp/venv/lib/python3.8/site-packages/tensorflow/core/framework/attr_value_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
File "/home/parklize/Documents/code/tfp/venv/lib/python3.8/site-packages/tensorflow/core/framework/tensor_pb2.py", line 16, in <module>
from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
File "/home/parklize/Documents/code/tfp/venv/lib/python3.8/site-packages/tensorflow/core/framework/resource_handle_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
File "/home/parklize/Documents/code/tfp/venv/lib/python3.8/site-packages/tensorflow/core/framework/tensor_shape_pb2.py", line 36, in <module>
_descriptor.FieldDescriptor(
File "/home/parklize/Documents/code/tfp/venv/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 561, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
The error occurred when I'm following in28minutes's course - "Microservices" on Udemy. When using maven build command "spring-boot:build-image -DskipTest" to create a Docker image, it gives the error. After digging a while, I found out that the default one is a socket without TCP on Ubuntu.
I had to allow Docker to accept requests from remote hosts by configuring it to listen on an IP address and port as well as the UNIX socket following steps from here.
sudo systemctl edit docker.service
Lets say we want to listen for connections from any remote host on port 2376
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376
sudo systemctl daemon-reload
sudo systemctl restart docker.service
sudo netstat -lntp | grep dockerd
here is the sequence:
## Calculate required sample size
def calc_sample_size(alpha, power, p, pct_mde, absolute=True):
""" Based on https://www.evanmiller.org/ab-testing/sample-size.html
Args:
alpha (float): How often are you willing to accept a Type I error (false positive)?
power (float): How often do you want to correctly detect a true positive (1-beta)?
p (float): Base conversion rate
pct_mde (float): Minimum detectable effect, relative to base conversion rate.
"""
if absolute:
delta = pct_mde
else:
delta = p*pct_mde
t_alpha2 = norm.ppf(1.0-alpha/2)
t_beta = norm.ppf(power)
sd1 = np.sqrt(2 * p * (1.0 - p))
sd2 = np.sqrt(p * (1.0 - p) + (p + delta) * (1.0 - p - delta))
return int(np.ceil((t_alpha2 * sd1 + t_beta * sd2) * (t_alpha2 * sd1 + t_beta * sd2) / (delta * delta)))
print(calc_sample_size(alpha=0.05, power=0.8, p=0.1, pct_mde=0.02))
Output:
3623