Photo by Annie Spratt on Unsplash
Why You Shouldn't Use Real Data for Development Purposes
It is not the right thing to do
It's not too uncommon that software development teams might end up using real production data instead of sample data for development purposes because of tight deadlines or budget constraints. In a worst-case scenario, the data is at risk of being sent over email in a rush, Skype, or shared on cloud-based platforms such as Slack or Discord.
This is not the right thing to do, it comes with a number of threats which might even propagate through your DevOps infrastructure.
The present guide shows step by step the implementation of a secure database design methodology that I call fixture-driven development. It consists in seeding a development database with sample fake data while designing it at the same time.
I used PHP and Symfony to write this tutorial but the important software design concepts can be easily transferred to other programming languages and frameworks too.
Before You Begin
Learn the basics of test-driven development
Install and set up a fresh copy of Symfony
Familiarize yourself with the concepts of fixtures and migrations
For further details please visit programarivm/zebra which is a GitHub repository where the present methodology was used to develop EasyAcl Bundle.
Fixture-Driven Development
A fixture-driven design methodology consists in letting the data fixtures guide the database design process in a similar way as with test-driven development (TDD).
So a fixture-driven cycle looks like this:
- Add a fixture
- Try to load the fixtures and see if they can be loaded
- Write some code in the entity layer
- Load the fixtures
- Repeat
Once the entire process is completed and therefore the database designed, you end up having precious sample data! Let's now look at how one of these cycles may look like.
1. Add a Fixture
Assuming the database design process is started completely from scratch, the very first fixtures need to be added. Possibly a good idea is to write the following ones in src/DataFixtures/UserFixtures.php
namespace App\DataFixtures;
use App\Entity\User;
use Doctrine\Bundle\FixturesBundle\Fixture;
use Doctrine\Common\Persistence\ObjectManager;
class UserFixtures extends Fixture
{
public function load(ObjectManager $manager)
{
$user = new User();
$user->setUsername('bob');
$manager->persist($user);
$manager->flush();
}
}
2. Try to Load the Fixtures
Of course at this point the Symfony console will throw an error if trying to load the user fixtures because the App\Entity\User
class does not exist yet.
$ php bin/console doctrine:fixtures:load
Careful, database "zebra" will be purged. Do you want to continue? (yes/no) [no]:
> yes
> purging database
> loading App\DataFixtures\AppFixtures
> loading App\DataFixtures\UserFixtures
2020-01-20T18:10:52+00:00 [critical] Uncaught Error: Class 'App\Entity\User' not found
In UserFixtures.php line 13:
Attempted to load class "User" from namespace "App\Entity".
Did you forget a "use" statement for another namespace?
This is an equivalent to making a test fail in a TDD methodology.
3. Write Some Code in the Entity Layer
So, the entity layer must be refactored with the purpose of making the previous fixtures to be loaded. A minimal basic src/Entity/User.php
file is written to achieve this goal.
namespace App\Entity;
use Doctrine\ORM\Mapping as ORM;
/**
* @ORM\Entity(repositoryClass="App\Repository\UserRepository")
*/
class User
{
/**
* @ORM\Id()
* @ORM\GeneratedValue()
* @ORM\Column(type="integer")
*/
private $id;
/**
* @ORM\Column(type="string", length=255)
*/
private $username;
public function getId(): ?int
{
return $this->id;
}
public function getUsername(): ?string
{
return $this->username;
}
public function setUsername(string $username): self
{
$this->username = $username;
return $this;
}
}
4. Load the Fixtures
The next step after adding the User
entity is to run the doctrine:migrations:diff
command in order to generate a migration file by comparing the current database to the mapping information.
$ php bin/console doctrine:migrations:diff
Generated new migration class to "/zebra/src/Migrations/Version20200120182410.php"
To run just this migration for testing purposes, you can use migrations:execute --up 20200120182410
To revert the migration you can use migrations:execute --down 20200120182410
Then the migration is run as described next.
$ php bin/console doctrine:migrations:migrate
Application Migrations
WARNING! You are about to execute a database migration that could result in schema changes and data loss. Are you sure you wish to continue? (y/n)yes
Migrating up to 20200120182410 from 0
++ migrating 20200120182410
-> CREATE TABLE user (id INT AUTO_INCREMENT NOT NULL, username VARCHAR(255) NOT NULL, PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8mb4 COLLATE `utf8mb4_unicode_ci` ENGINE = InnoDB
++ migrated (took 79.2ms, used 12M memory)
------------------------
++ finished in 87.6ms
++ used 12M memory
++ 1 migrations executed
++ 1 sql queries
And finally the fixtures are loaded again.
$ php bin/console doctrine:fixtures:load
Careful, database "zebra" will be purged. Do you want to continue? (yes/no) [no]:
> yes
> purging database
> loading App\DataFixtures\AppFixtures
> loading App\DataFixtures\UserFixtures
This is how the user
table will look like after the first fixture-driven development cycle is completed.
mysql> select * from user;
+----+----------+
| id | username |
+----+----------+
| 1 | bob |
+----+----------+
1 row in set (0.00 sec)
5. Repeat
It is time to prepare the next cycle. On this occasion probably it is a good thing to keep on adding properties to the User
entity.
namespace App\DataFixtures;
use App\Entity\User;
use Doctrine\Bundle\FixturesBundle\Fixture;
use Doctrine\Common\Persistence\ObjectManager;
class UserFixtures extends Fixture
{
public function load(ObjectManager $manager)
{
$user = new User();
$user->setUsername('bob')
->setEmail('bob@gmail.com');
$manager->persist($user);
$manager->flush();
}
}
As expected, the loading of the fixtures will now fail because the setEmail()
method doesn't still exist in App\Entity\User
.
$ php bin/console doctrine:fixtures:load
Careful, database "zebra" will be purged. Do you want to continue? (yes/no) [no]:
> yes
> purging database
> loading App\DataFixtures\AppFixtures
> loading App\DataFixtures\UserFixtures
2020-01-20T19:34:14+00:00 [critical] Uncaught Error: Call to undefined method App\Entity\User::setEmail()
In UserFixtures.php line 15:
Attempted to call an undefined method named "setEmail" of class "App\Entity\User".
Conclusion
This post is to encourage developers to write a suite of fixtures to be loaded in a testing database, it is highly discouraged to use real production data for development. The fixtures are written while designing the database.
The first fixture-development cycle described above was pretty straightforward for the time being; at some point after future cycles of development the App\Entity\User
entity might end up resembling something as it is shown next.
namespace App\DataFixtures;
use App\Entity\User;
use Doctrine\Bundle\FixturesBundle\Fixture;
use Doctrine\Common\Persistence\ObjectManager;
use Faker\Factory;
use Symfony\Component\Security\Core\Encoder\UserPasswordEncoderInterface;
class UserFixtures extends Fixture
{
const N = 20;
private $encoder;
private $faker;
public function __construct(UserPasswordEncoderInterface $encoder)
{
$this->encoder = $encoder;
$this->faker = Factory::create();
$this->faker->addProvider(new \Faker\Provider\Internet($this->faker));
}
public function load(ObjectManager $manager)
{
for ($i = 0; $i < self::N; $i++) {
$user = new User();
$user->setUsername($this->faker->username)
->setEmail($this->faker->email)
->setPassword($this->encoder->encodePassword(
$user,
$this->faker->password
));
$manager->persist($user);
$this->addReference("user-$i", $user);
}
$manager->flush();
}
}
mysql> select * from user;
+----+------------------+-----------------------------+---------------------------------------------------------------------------------------------------+
| id | username | email | password |
+----+------------------+-----------------------------+---------------------------------------------------------------------------------------------------+
| 1 | demarcus.koch | cristal.hagenes@treutel.com | $argon2id$v=19$m=65536,t=4,p=1$LmX8EoXrkemR8VLIO1ZcTA$5UC0tWn+6fJEma/neKcitgpELZlPTAcL9wC6O1nNfn8 |
| 2 | ablanda | floy73@marks.com | $argon2id$v=19$m=65536,t=4,p=1$Iq5mhGD4M5bYEvRcszqdsA$9sJExnkdKO2QMR8PA+DFZ+OL8EvCkvJ1WdhkUMuELN8 |
| 3 | ernesto.wolf | zstanton@hotmail.com | $argon2id$v=19$m=65536,t=4,p=1$0mV3Wg6DADHoA8XjXu555w$RljkrrtDABBOTkQSNNmBOKA16e2jAcwRnBs6qjOIPhQ |
| 4 | bwhite | gsmith@gmail.com | $argon2id$v=19$m=65536,t=4,p=1$7u2ecWB6IedLaMHY+bhb6g$72VHnkYwZw458orExJvvEaoZMLcHoL4f6upn+73qJZ8 |
| 5 | dorris28 | vgrimes@hotmail.com | $argon2id$v=19$m=65536,t=4,p=1$4+sjf5jOIkNPrgagmsCoMw$qxDBCxNy16ODcPAoDjbo0AgS2TnM/QGxo8kEIqWugSI |
| 6 | xrodriguez | ledner.verla@franecki.com | $argon2id$v=19$m=65536,t=4,p=1$YhR3tHoopKSJNhehtZ4sIg$jxxKrrT9R6Weg4IbcSEdHn1WQoNmT2/OYuRfTaGTjjY |
| 7 | rcronin | ngutkowski@yahoo.com | $argon2id$v=19$m=65536,t=4,p=1$TKtVQCp4chLWbhiGR+/axA$TbmnyuEmWIdtq5yNzPbjOlYeQNH3tFa612+LYwVxc3Y |
| 8 | wunsch.edwardo | freda16@wiza.com | $argon2id$v=19$m=65536,t=4,p=1$Pp9AtCXw/XB6/Z5FiFH3aA$X420Ix4JAGcRu0x2gJdoToVeXSSFlZU33BeDR13/gOQ |
| 9 | jacobi.broderick | ortiz.declan@gmail.com | $argon2id$v=19$m=65536,t=4,p=1$31b2sF9Gt7WfAdpdLGofLg$PV/T4vCb9NUwprkKCS1N/MFSopHXFLiaGSxRMS/TtKw |
| 10 | halie.beier | dsimonis@haley.com | $argon2id$v=19$m=65536,t=4,p=1$FW5MHfVG7evZan5uEEZhUA$S/tlt4LocrW998oDrW3K5k/LE5JUY0ioFMxgICg0pC4 |
| 11 | ortiz.corine | ostoltenberg@yahoo.com | $argon2id$v=19$m=65536,t=4,p=1$Vr2hEI+PH6QPBY1elab5cg$wTM9Tkd4IcAyF9Eo0Oxec/ejlC/LgZg5oRtp39gICSQ |
| 12 | uklocko | emelia.donnelly@hyatt.org | $argon2id$v=19$m=65536,t=4,p=1$4TvtIIJ1Lf4vHEjnZ+qubg$kOin2zZ40wWCRn3Wkn9U2X9ijmpQJ69duk/Yfh4WnOg |
| 13 | williamson.willy | ikeeling@tillman.com | $argon2id$v=19$m=65536,t=4,p=1$UdUPCVJFIM2xLtIHMW8iqg$ucx9nuPuS92GXfU1WQGew8YjTBhlJURk7vgzwIZadXk |
| 14 | schoen.tristin | thompson.zion@mante.com | $argon2id$v=19$m=65536,t=4,p=1$DvvUHieQhWKFLNZ98NzWJw$QMd3G6zhW9uEPttwXpMk6DMvkr4TrDdFEBLgv3GpV3w |
| 15 | collier.xavier | emelie.padberg@yahoo.com | $argon2id$v=19$m=65536,t=4,p=1$hGH8E+gf6fz8IoNJHaFiGA$M6euacVeiTgT5cVHvPctx3cTInWCRfk5b2rX50imOwg |
| 16 | maxine.purdy | ibednar@herman.info | $argon2id$v=19$m=65536,t=4,p=1$VrFDOh6XF7bsXkUNyGYHzg$is+BNG3ZyxE0IEazy6ZNmdV/9n3omhgRXpQY4aH63OA |
| 17 | anabelle50 | jmoen@gmail.com | $argon2id$v=19$m=65536,t=4,p=1$ad/+ztC2VYgoFncfnKvd7w$Swqo2Wg0lHqObIEMRdkI4lwjAIDTG9hy1J30pUa8MGs |
| 18 | titus.damore | qlowe@hotmail.com | $argon2id$v=19$m=65536,t=4,p=1$1hUK0xzB1iykjHtGx3cRpA$A/sXhRQwnnzSJ46NFW2te9xO84qs+WW5Y4wFQGdeSL8 |
| 19 | amy.cronin | kaitlyn66@toy.biz | $argon2id$v=19$m=65536,t=4,p=1$RTzEBBGF+vrL8heH9I2CMw$bMSFOkbCapen1+3+feJmWQ/77hdjiHShrjbVudTxN9s |
| 20 | darryl64 | friedrich00@feeney.com | $argon2id$v=19$m=65536,t=4,p=1$dYpnao4o5MXhLEgzY82Q5A$hZ+4JR7/2ZjW2WLG1HJa8mTNFnBWLrYmDj4SuuhWCwc |
+----+------------------+-----------------------------+---------------------------------------------------------------------------------------------------+
20 rows in set (0.00 sec)
Did you find this article valuable?
Support Jordi Bassaganas by becoming a sponsor. Any amount is appreciated!