Pre-heat the oven to 320 and line two 9 inch cake pans with parchment. Spray the cake pans well with unflavored cooking spray. Combine the flour and baking powder in a bowl and set it aside. Then set up a stand mixer and fit it with the whisk attachment. Combine the sugar and butter in its bowl and beat them together until fluffy. Add in the eggs one at a time, followed by the vanilla. Let it all get whipped up on a higher speed for a minute, then turn off the mixer. Pour the dry ingredients into the bowl through a fine mesh strainer so that it is very fine and light. Fold it in with a spatula. Then fold in the lemon zest, lavender and the milk to moisten it. Divide the batter evenly among the two prepared cake pans and bake the layers for 25 to 30 minutes. Do not open the oven door or turn on the light until 25 minutes in to check them. Let them cool completely. 

While the cakes bake and cool, make the earl grey syrup. Combine the water, sugar, tea bag and vanilla in a small saucepan and heat it over medium high heat. Let it come to a gentle boil while you stir it. Once it starts to coat the spoon and thickens into a syrup, take it off the heat to cool. Then make the whipped cream. Combine the heavy cream, sugar and 1 teaspoon of the lemon zest in a mixing bowl and use a hand mixer to whip it up into stiff peaks. It should be airy but not runny at all. Then it is time to assemble this baby!  

Place the first layer flat side up on a cake plate and poke holes all over it. Drizzle in half the syrup to soak into the sponge. Then spread half the whipped cream on it evenly. Place the second layer on flat side down. Poke holes and drizzle the remaining syrup into the sponge. Spread the remaining whipped cream on top, then sprinkle the remaining lemon zest and the ground lavender on top. You could also decorate with lemon wedges or candies lemon peel! Serve and enjoy. It also keeps for 2 days covered in the refrigerator!