Finished converting to XNA 4.0

At last I have done the final bit so that I can say I have everything in XNA 4.0 that I had in XNA 3.1

This final part was not a code change to the game but is a way to get animations from my models created in Blender in to XNA. In XNA 3.1 the FBX importer could read multiple takes from one FBX file and I took advantage of that to import all the animations. That is no longer built in to the FBX importer.

I am rather pleased with my solution. It is much more elegant than before. I have created myself a WinForm based application that can read FBX files created from Blender.  It can split them in to one file per take and export the animation key frames in to the format I need.

Compared to games it is not a very pretty presentation.  Functional with a text status window.

I can create a config file per model so that the extraction, conversion and renaming of takes, to match my in game requirements, is reproducible.  I should probably have done it this way in the first place.

If you are interested the source code can be downloaded from:
http://code.google.com/p/take-extractor/

I can now remove a lot of redundant code from my game editor.

Edit: I spoke too soon.  The animations do not work properly from XNA 4.0 so I still need an XNA 3.1 utility just to get the animations right.  The game will still be in XNA 4.0!  I'm trying to find an alternate exporter for Blender or other method to be able to use only XNA 4.

Anti-Aliasing with XNA 4

I've spent all weekend struggling with getting Anti-Aliasing (AA) working with XNA 4.  I compared my old XNA 3.1 version with my updated XNA 4 version and noticed that the graphics on the menus looked awful on the new one!

This was obviously because AA, multi-sampling, was not enabled on the new version.  Some of the settings used to enable this in XNA 3.1 had been replaced in XNA 4.  I therefore spent a lot of the weekend on the assumption that I just needed to get the settings right within XNA.

I asked for help on the XNA forums so you can read the saga there as well.

To demonstrate the problem, the following rather rough picture is a small portion of the screen illustrating the effect:

The following much smoother shot shows what I was expecting:

To cut a long frustrating story short.  On advice I created a small app that had nothing but one model and a background and that helped to narrow the problem down.  It was not within the XNA code but it was in the HLSL shader code.

These three lines turn off anti-aliasing in XNA 4.0 and do nothing in XNA 3.1!
MinFilter = Point;
MagFilter = Point;
MipFilter = Point;
 
I removed those three lines and everything was back to what I expected.  This is the end result:
 

I've gone a little further and checked out the other options for the texture filters when set within the HLSL fx file.  I got the options from the following web page:
http://msdn.microsoft.com/en-us/library/bb322811(VS.85).aspx

The results are:
Min, Mag and Mip filters = Point -> AA gets turned off
Min, Mag and Mip filters = Linear -> AA works
Min and Mag filters = Anisotropic -> AA works
Mip filter = None -> AA works but the result was grainy
Min and Mag filters = PyramidalQuad -> Failed to compile
Min and Mag filters = GaussianQuad -> Failed to compile

My conclusion is that I should remove everything from my HLSL effect files that can now be set within XNA 4.0 and I should set those from within XNA.  As always one of Shawn Hargreaves blog posts is most useful for that.

[Edit:  change of plan again.  I completely re-wrote my shaders and now the filtering is in XNA code... again.]

More XNA 4 stuff

Over the weekend and still continuing the XNA forums are down for an upgrade.  Everyone suspects this is because of the Windows Phone 7 launch.

This is inconvenient and I have had to resort to Twitter for advise.  Not as bad as I thought everyone was very helpful.  Thanks to @mikebmcl, @UberGeekGames and others. 140 characters is a bit short to get much across but I got what I needed to know.

I had to re-write the muzzle flash code to make it easier to change the graphic.  My previous code, working in XNA 3.1, produced a too harsh effect in XNA 4.  I wanted it to fade out a bit more at the edges.  In the process I discovered that the new XNA 4 Particle3D sample on the creators club uses the XNA 3.1 syntax for VertexDeclarations.  It did not stop it working but I like to use the best methods so I re-wrote those as well.

I have three major areas left to get working properly.  Shadows, which are now very rough on Windows because of the enforced HLSL instruction slot limits, 512 for PS_3_0.  As the game is for the Xbox360 I am not going to worry about that until I see the results on the 360.

Animations are limited to only one per FBX file.  I am also not worried about this because at worst I could export all my animations from Blender one at a time.  I will most likely change the code to simply use the FBX file almost unchanged instead of my own animation format only invented in order to save the animations individually.

What I am left with to deal with are triangles...  The new model class has a separate VertexBuffer for each mesh.  The old one had one vertex buffer for the whole model.  In theory I should find this easier to extract triangles from each model.  My first quick guess did not work.

I need to reinstate my debug code that lets me view the triangles so that I know when I have got it right.

In game the collision is based on spheres but the editor uses the triangles to calculate the location of the spheres.  In game the only time the individual triangles are used is to calculate the orientation of bullet impact decals so that they line up correctly with the faces of the model.

I'd better get on with it instead of typing this.

[Edit]
I did what I said and the resulting code to extract triangles can be found on the following web page:
http://www.discoverthat.co.uk/games/xna.htm#vertex

I found another similar blog giving yet more useful details on the transition from XNA 3.1 to 4.0 it covers much of the same ground as my posts but also has some example code:
http://blogs.microsoft.co.il/blogs/grozen/archive/2010/10/23/converting-a-3d-xna-3-0-game-to-xna-4-0-an-example.aspx

XNA 4.0 from XNA 3.1

Last night my code finally compiled in XNA 4.0.  I had reached a good point last Friday and thought I had some time so I decided then was a good time to upgrade.  For those reading this who are not familiar with XNA.  It is the game development framework that Microsoft has provided for Indie and hobby developers.  It is based on C# and is the only way for hobby programmers to release games on the Xbox 360.

XNA 4.0 is the latest version.  It has some improvements but also some pain converting from the older version.  In all it has taken me about 20 hours to get this far.

As you can see from that picture it is not right.  Most of the mess is caused by the change to defaulting to precomputed alpha.  Although I have changed my code I still need to change the graphics or the way I use them to suit that.  Something new for me to learn.

The unexpected problem is that the model of the space ship is showing it's insides on the outside.  I'm sure I will work out how to fix it but at the moment it is a bit frustrating!

What I wanted to write here was just a list of all the things I have had to do to convert from XNA 3.1 to XNA 4.0.  Partly to let off steam and hopefully to help some other people doing the same.  I've included some useful links as well.

- The conversion to XNA 4.0 did not get all the references right.
I had to manually remove them and re-add them.  This cannot be done from within the IDE it was necessary to edit the project file with a text editor.  Very quick and painless.

Remove and re-add References.  Most can be added back using Visual Studio but a couple have to be manually edited:
mscorlib (can not add back with VS edit .csproj file)
System
System.Core (can not add back with VS edit .csproj file)
System.Xma
System.Xml.Linq

To add back missing items see the following post:
http://social.msdn.microsoft.com/Forums/en/windowsphone7series/thread/07af6fac-7b07-4dca-b9de-7e18c01d5f0b

The .csproj file needs to be edited manually using a text editor.  Remember to close Visual Studio first.

The following image shows the syntax for the .csproj file:

- I've changed all the colours where I set the alpha to the new syntax.
I have not, as yet, changed the graphics to match.  Hence the mess in the above picture.
http://blogs.msdn.com/b/shawnhar/archive/2010/04/08/premultiplied-alpha-in-xna-game-studio-4-0.aspx
Changed new Color(colour, alphaByte) [0 to 255] to colour * alphaFloat [0.0 to 1.0]

- Several changes to the shader effect files
AdderssU and V only support: Wrap, Clamp and Mirror.
Border and BorderColor are gone.
http://blogs.msdn.com/b/shawnhar/archive/2010/04/02/state-objects-in-xna-game-studio-4-0.aspx
Set all the Mip, Mag and Min filters on the shadowmaps to Point.  Not sure why but unless I do that I get an exception when running the game.
The new effect shader compiler is also much fussier on Windows.  It enforces the minimum number of instruction slots for each shader model, e.g. PS 3 can have a minimum of 512 instruction slots.  My graphics card could have more but the compiler does not let me use them.  The Xbox 360 compiler is virtually unchanged and so my old shader code compiled for the Xbox but not for Windows.  I've had to reduce the quality of the shadows on Windows to fit within the 512 limit in one pass.  I've added a compiler directive #if defined(XBOX360) so I can do that.

- Re-wrote the way the sky is drawn.  The new Creators Club sample no longer uses a separate effect file.
http://creators.xna.com/en-GB/sample/generatedgeometry
[Edit: had to re-instate the old version because the new one did not work with my terrain!] 

- Changed the exception in the Skinned model prosessor to a warning about not having any animations.
My models already have separate animations so the skinned models need to be loaded even though they do not have any animations included with them.
 
- I have temporarily removed my built in screen shot class.  Resolve back buffer has been removed and it is now necessary to use a RenderTarget to get the screenshot.  As this puts unnecessary code in  to my draw loop I have not done it yet.
The screenshots are only for my benefit and they were not going to be in the finished game so it's no loss.
http://blogs.msdn.com/b/shawnhar/archive/2010/03/30/resolvebackbuffer-and-resolvetexture2d-in-xna-game-studio-4-0.aspx

http://blogs.msdn.com/b/shawnhar/archive/2010/03/26/rendertarget-changes-in-xna-game-studio-4-0.aspx
http://forums.xna.com/forums/t/55428.aspx

- Removed all Depth Stencil buffers and added back the type in the RenderTarget2D where necessary.
Reading about this was more confusing than just doing it.  The new XNA 4.0 method is much easier.
http://blogs.msdn.com/b/shawnhar/archive/2010/03/16/breaking-changes-in-xna-game-studio-4-0.aspx

http://www.innovativegames.net/blog/blog/2010/07/13/xna-4-0-breaking-changes-rendertargets/
http://blogs.msdn.com/b/shawnhar/archive/2010/03/26/rendertarget-changes-in-xna-game-studio-4-0.aspx

- VertexDeclarations are also no longer needed.
http://blogs.msdn.com/b/shawnhar/archive/2010/04/19/vertex-data-in-xna-game-studio-4-0.aspx
All you need is: graphicsDevice.SetVertexBuffer(vertexBuffer);
Lots of lines deleted.

- The syntax for drawing user primitives has been simplified
Most of my code now just reads something similar to the following, instead of all the Begin() and End() methods, just:
basicEffect.CurrentTechnique.Passes[0].Apply();
http://www.bit-101.com/blog/?p=2832

- Replaced my particle system with a version based on the new sample because the old sample was based on point sprites which are no longer supported.
http://creators.xna.com/en-GB/tutorial/particlexml
Fairly easy change but as yet untested.
[Edit: needed further work to get it working nicely.]

- Changed every SpriteBlendMode.AlphaBlend to BlendState.AlphaBlend.
There were a lot of them in my code but easy to find and replace.

- On one page I used a Scissor test and the syntax has change to use RasterizerState.ScissorTestEnabled

- The Texture load and save functions have been improved.
I have changed all my BMP files to now use PNG.
http://blogs.msdn.com/b/shawnhar/archive/2010/05/10/image-codecs-in-xna-game-studio-4-0.aspx
[Edit: this saved as pre-multipled alpha which was not what I needed so I have had to change one file format.]

- Lots of changes on the storage side of things.
There is no TitleLocation anymore.
I have had to use the Manifest example for some lists:
http://creators.xna.com/en-GB/sample/contentmanifestextensions
I have changed how the StorageContainer is accessed for saving user files:
http://blogs.msdn.com/b/nicgrave/archive/2010/07/23/storage-in-xna-game-studio-4-0.aspx
For my editor I have used .NET System.IO file methods and a semi hard coded path to the same place as the user storage location in Windows.  This is because the Path attribute from the StorageContainer has been removed.
I could have used any location for the editor but that one keeps everything in one known place.
A useful method is:
System.Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments)

- The syntax for turning on Antialising has changed or at least the method I was using has gone.  The method I now use is:
graphicsDevice.PreferMultiSampling = true;
[Edit: this needed a lot more work see the future post.]

That's what I have done so far.  The game works but looks awful so I still have lots more work to do to get back to where I was with the older version a week ago!

[Edit:  After I did my conversion I found better tutorials: http://www.nelxon.com/blog/xna-3-1-to-xna-4-0-cheatsheet/ ]

Get the triangles at run time from a model

The following is based on various other code and was produced with assistance from the XNA forums.

Download Source Code

Vertex Helper XNA 4.0 class (3kb) Oct. 2010

The following reads the vertex buffer of each model mesh part and stores the triangles in a vertex and index array.

public class VertexHelper
{
 public VertexHelper()
 {
 }

 public struct TriangleVertexIndices
 {
  public int A;
  public int B;
  public int C;
 }

 /// 

 /// Extract the vertices and indices from the specified model
 /// 
 /// Output the list of vertices
 /// Output the list of indices
 /// The models world position or use Matrix.Identity for object space
 public static void ExtractTrianglesFrom(Model modelToUse, List vertices, List indices, Matrix worldPosition)
 {
  Matrix transform = Matrix.Identity;
  foreach (ModelMesh mesh in modelToUse.Meshes)
  {
   // If the model has bones the vertices have to be transformed by the bone position
   transform = Matrix.Multiply(GetAbsoluteTransform(mesh.ParentBone), worldPosition);
   ExtractModelMeshData(mesh, ref transform, vertices, indices);
  }
 }

 /// 

 /// Transform by a bone position or Identity if no bone is supplied
 /// 
 public static Matrix GetAbsoluteTransform(ModelBone bone)
 {
  if (bone == null)
  {
   return Matrix.Identity;
  }
  return bone.Transform * GetAbsoluteTransform(bone.Parent);
 }

 /// 

 /// Get all the triangles from all mesh parts
 /// 
 public static void ExtractModelMeshData(ModelMesh mesh, ref Matrix transform,
  List vertices, List indices)
 {
  foreach (ModelMeshPart meshPart in mesh.MeshParts)
  {
   ExtractModelMeshPartData(meshPart, ref transform, vertices, indices);
  }
 }

 /// 

 /// Get all the triangles from each mesh part (Changed for XNA 4)
 /// 
 public static void ExtractModelMeshPartData(ModelMeshPart meshPart, ref Matrix transform,
  List vertices, List indices)
 {
  // Before we add any more where are we starting from
  int offset = vertices.Count;
  
  // == Vertices (Changed for XNA 4.0)

  // Read the format of the vertex buffer
  VertexDeclaration declaration = meshPart.VertexBuffer.VertexDeclaration;
  VertexElement[] vertexElements = declaration.GetVertexElements();
  // Find the element that holds the position
  VertexElement vertexPosition = new VertexElement();
  foreach (VertexElement vert in vertexElements)
  {
   if (vert.VertexElementUsage == VertexElementUsage.Position &&
    vert.VertexElementFormat == VertexElementFormat.Vector3)
   {
    vertexPosition = vert;
    // There should only be one
    break;
   }
  }
  // Check the position element found is valid
  if (vertexPosition == null || 
   vertexPosition.VertexElementUsage != VertexElementUsage.Position ||
   vertexPosition.VertexElementFormat != VertexElementFormat.Vector3)
  {
   throw new Exception("Model uses unsupported vertex format!");
  }
  // This where we store the vertices until transformed
  Vector3[] allVertex = new Vector3[meshPart.NumVertices];
  // Read the vertices from the buffer in to the array
  meshPart.VertexBuffer.GetData(
   meshPart.VertexOffset * declaration.VertexStride + vertexPosition.Offset, 
   allVertex, 
   0, 
   meshPart.NumVertices,
   declaration.VertexStride);
  // Transform them based on the relative bone location and the world if provided
  for (int i = 0; i != allVertex.Length; ++i)
  {
   Vector3.Transform(ref allVertex[i], ref transform, out allVertex[i]);
  }
  // Store the transformed vertices with those from all the other meshes in this model
  vertices.AddRange(allVertex);

  // == Indices (Changed for XNA 4)

  // Find out which vertices make up which triangles
  if (meshPart.IndexBuffer.IndexElementSize != IndexElementSize.SixteenBits)
  {
   // This could probably be handled by using int in place of short but is unnecessary
   throw new Exception("Model uses 32-bit indices, which are not supported.");
  }
  // Each primitive is a triangle
  short[] indexElements = new short[meshPart.PrimitiveCount * 3];
  meshPart.IndexBuffer.GetData(
   meshPart.StartIndex * 2, 
   indexElements, 
   0, 
   meshPart.PrimitiveCount * 3);
  // Each TriangleVertexIndices holds the three indexes to each vertex that makes up a triangle
  TriangleVertexIndices[] tvi = new TriangleVertexIndices[meshPart.PrimitiveCount];
  for (int i = 0; i != tvi.Length; ++i)
  {
   // The offset is because we are storing them all in the one array and the 
   // vertices were added to the end of the array.
   tvi[i].A = indexElements[i * 3 + 0] + offset;
   tvi[i].B = indexElements[i * 3 + 1] + offset;
   tvi[i].C = indexElements[i * 3 + 2] + offset;
  }
  // Store our triangles
  indices.AddRange(tvi);
 }


}

A version of the above working in XNA 3.1 is available from: enchantedage.com